JEP 181: Align JVM Checks with Java Language Rules for Nested Classes

OwnerJohn Rose
Created2013/03/19 20:00
Updated2017/05/02 17:04
Componenthotspot / runtime
Discussionvalhalla dash dev at openjdk dot java dot net


Align the JVM access checking rules with Java language rules for methods, constructors, and fields in nested classes, by partitioning classes into nests -- related classes that share a common access control context (usually derived from the same source file). In particular, allow a class file to access private names of other class files compiled within the scope of a single top-level declaration without the need for compilers to insert accessibility-broadening bridge methods.


Provide VM support for allowing language compilers to group related classes into a nest, which share an access control context. This allows classes that are compiled to distinct classfiles but are logically part of the same code entity (such as inner classes in Java) to access each others members without the need for artifacts such as access bridges.

Ensure that class files contain accurate descriptions of class and interface nesting.

Provide necessary VM foundation work for desired future features, such as a safe and supported alternative to Unsafe.defineAnonymousClass(), and sealed classes.


This JEP is not concerned with large scales of access control, such as modules.


Many JVM languages support multiple classes in a single source file (such as Java's nested classes), or translate non-class source artifacts to classfiles. From a user perspective, however, these are generally considered to be all in "the same class", and therefore users expect them to share a common access control regime. To preserve these expectations, compilers frequently have to effectively lower the access of private members to package, through the addition of access bridges, but these bridges subvert encapsulation and can confuse users and tools. A formal notion of a group of classfiles forming a nest, where nest mates share a common access control mechanism, allows the desired result to be achieved in a simpler, more secure, more transparent manner.

The notion of a common access control context arises in other places as well, such as the host class mechanism in Unsafe.defineAnoymousClass(), where a dynamically loaded class can use the access control context of a host. A formal notion of nest membership would put this mechanism on firmer ground (but actually providing a supported replacement for defineAnonymousClass() would be a separate effort.)


The Java Language Specification allows classes and interfaces to be nested within each other. Within the scope of a top-level declaration (JLS 7.6), any number of types can appear nested. We can describe a top-level type, plus all types nested within it, as forming a nest, and two members of a nest are described as nestmates. Nestmates have unrestricted access to each other (JLS 6.6.1), including to private fields, methods, and constructors. The private access is complete (undifferentiated, flat) within the whole declaration of the containing top-level type. (One can think of this as a top-level type defining a sort of "mini-package", within which extra access is granted, even beyond that provided to other members of the same Java package.)

The Java compiler compiles a group of nested types into a corresponding group of class files; it uses the InnerClasses and EnclosingMethod attributes to reify the nesting relationship (JVMS 4.7.6 and 4.7.7). While these attributes are enough for the JVM to determine nest-mate-ness, to allow for a broader, more general, notion of nestmates beyond simply Java language nested types, and for the sake of efficiency, it is proposed to modify the class file format to define two new attributes for use by nestmates and top-level classes (known as the nest top). Each nestmate has an attribute to identify its nest top class; and each nest top class contains an attribute to identify known nest members.

We will adjust the JVM's access rules by adding something like the following clause to JVMS 5.4.4:

A field or method R is accessible to a class or interface D if and only if any of the following conditions are true:

  • ...
  • R is private and is declared in a different class or interface C, and C and D, are nestmates.

For types C and D to be nestmates they must either have the same nest-top. A type C claims to be a member of the nest of D, if it lists D in it's MemberOfNest attribute. The membership is validated during class loading if D also lists C in its NestMembers attribute. This may require loading of nest-top types earlier than might otherwise occur.

With this change to the access rules, and with suitable adjustments to the invokeSpecial bytecode (and possibly others), the javac wrappers are no longer needed and javac can generate direct member access instructions for private nestmate members.

By codifying the notion of nestmates and the associated access rules within the VM, we not only allow javac to shed that role and tighten the existing security checks, but also allow for future enhancements to take advantage of this notion. For example:

The loosened access rules would affect access checks during the following activities:

It would also require adjustment to other constraints currently imposed on the use of the invokespecial bytecode, for example:

and on the semantics of MethodHandle invocation (which mirrors the invokespecial constraints). Non-prescriptive language may need updating, such as in JVMS 3.7.

By opening up private nestmate access at the VM level, we can also restrict the other accessors generated by javac to be private rather than the current package-private. In many cases, the accessors can be eliminated altogether, or replaced by shared-but-private method handle constants.

Open Issues

  1. This proposal widens access for nestmates, at the JVM level. While we are at it, should we narrow access to classes which are declared protected or private, to more closely match the Java language rules? This would require the JVM to perform an additional access check based on the value of Class.getModifiers. (Probably not, since this could break loosely-written reflective code if it assumes private access is weakened to package-private. Also, new checks on protected classes could cause global effects, since they are rendered to the JVM as public classes.)

  2. Should nestmate-ship be exposed via core reflection? Core reflection will need to be adjusted for nestmate access rules, but it remains to be determined whether we should have an explicit public API to query nestmate relationships. A simple query function would do it: Class#getNestTop which returns null if the class is not part of a nest, else the top class of the nest. (If the class itself is the top, the query returns the class itself.)


We can continue generating wrapper methods in the Java compiler, as needed. This is a hard process to predict. For example, Project Lambda had difficulty resolving method handle constants in the presence of inner classes, leading to a new type of bridge method. Because compiler-generated wrappers are tricky and unpredictable, they are also buggy and hard to analyze by various tools, including decompilers and debuggers.

The initial proposal considered using the existing InnerClasses and EnclosingMethod attributes to establish nestmate-ship. But introducing specific nestmate related attributes both makes nestmates more general than only relating to language-level nested types, and permits a more efficient implementation.


We will need an extensive set of JVM tests to verify the new access rules and adjustments to the bytecode semantics to support nestmates.

Similarly we will need additional tests for core reflection, method handles, var-handles, and external access API's like JDWP, JVM TI and JNI.

Since no language tests are proposed here, no new language compliance tests are needed.

Adequate functional tests for will arise naturally from language compliance tests, after the Java compiler is modified to rely on nestmate access.

Risks and Assumptions

The new rules would have to be associated with a new class file version number, since the rules for Java compilers would change.

Java compilers would be required to retain wrapper-generation logic for backward compatibility with older target JVMs.

Loosening access presents little or no conformance risk. Exception: Negative compliance tests could fail, in principle, but this seems unlikely.

There is little or no risk to user compatibility, since the proposal loosens access. If users have "discovered" and exploited wrapper methods, they will be unable to do so after the wrappers are dropped. Such risk is very small, since wrapper methods do not have stable names in the first place.

There is little or no risk to system integrity, since the proposed rules confer new access only within a single runtime package. By removing the need for wrapper methods, potential access between distinct top-level classes will be systematically decreased.


This change will require new language in the JVM specification, as well as changes to the JVM implementation. It will also require changes to the specifications and implementations of core reflection, method handles, var-handles, and potentially JVM TI, JDWP and JNI (though given the native interfaces tend to ignore access controls in general, there may not be much to do).

The extra complexity of access checking is something that has to be examined, as will the switch from use of invokevirtual of an accessor, to invokespecial of a private method.

The Java compiler currently generates access bridges for cross-nest access; while it is not strictly required to remove these immediately as part of this feature, once such bridges become unnecessary, it is desirable to have the compiler no longer generate them.

The rules for mapping Java source constructs to class files will be simplified. This is especially timely, since Project Lambda is complicating the same rules. Some cross-product effects have been observed (JDK-8005122), so the recent increase in complexity is not simply additive.

Dropping access methods will slightly decrease the size of some applications.

The Pack200 specification may need adjustment.