JEP 12: Incubating Language and VM Features

OwnerAlex Buckley
Created2018/01/19 01:27
Updated2018/02/05 19:33
TypeInformational
StatusProposed to Target
ScopeSE
Discussionjdk dash dev at openjdk dot java dot net
EffortM
DurationM
Priority2
Reviewed byAlan Bateman, Brian Goetz, Mark Reinhold
Endorsed byMark Reinhold
Issue8195734

Summary

An incubating language or VM feature is a new feature of the Java SE Platform that is fully specified, fully implemented, and yet impermanent. It is available in a JDK feature release to provoke developer feedback based on real world use; this may lead to it becoming permanent in a future Java SE Platform.

Goals

Non-Goals

Motivation

The Java SE Platform has global reach, so the cost of a mistake in the design of a Java language or VM feature is high. ("VM feature" encompasses the class file format and the JVM bytecode instruction set.) A mistake may be a hard technical error (such as a flaw in the type system), a soft usability problem (such as a surprising interaction with an older feature), or a poor architectural choice (such as one that forecloses on directions for future features).

To build confidence in the correctness and completeness of a new language or VM feature, it is desirable for the feature to enjoy a period of broad exposure after its specification and implementation are stable but before it achieves final and permanent status in the Java SE Platform. This period is called incubation. Including the feature in a mainline JDK feature release will give the broadest possible exposure and opportunity for swift feedback. It will also encourage tool vendors to build good support for the feature before the bulk of Java developers use it in production. Before the next JDK feature release, the feature's "real world" strengths and weaknesses will be evaluated to decide if the feature has a long-term role in the Java SE Platform and, if so, whether it needs refinement. Consequently, the feature may be granted final and permanent status (with or without refinements), or undergo a period of "re-incubation" (with or without refinements), or else be removed.

Incubation of language and VM features is distinct from incubation of APIs, because APIs can be delivered in discrete incubator modules whereas language and VM features can only be delivered in the compiler and runtime system.

Description

Design of an Incubating Feature

An incubating language or VM feature is a new feature whose design, specification, and implementation are all complete, but which would benefit from a period of broad exposure and evaluation before either achieving final and permanent status in the Java SE Platform or else being refined or removed.

An incubating feature must meet the high standards that are expected for any feature of the Java SE Platform. For example, an incubating feature must respect traditional Java principles such as readability and compatibility, and it must receive appropriate treatment in the reflective and debugging APIs of the Java SE Platform. In other words, an incubating feature is expected to display the same level of technical excellence and finesse as a feature that has already achieved final and permanent status in the Java SE Platform. As a general rule, for anything not explicitly spelled out in this JEP, the policy is "No different for an incubating feature than for any other feature."

The Umbrella JSR for the Java SE $N Platform enumerates the incubating language and VM features in the platform. The desire for feedback and the expectation of quality means that these features are not "optional"; they must all be supported in full by every implementation of the Umbrella JSR. They are specified in appendices of the Java SE $N Editions of the Java Language Specification (JLS) and the Java Virtual Machine Specification (JVMS). The Java Compatibility Kit for Java SE $N checks conformance of an implementation's incubating features to the appropriate appendix.

Incubation is not a mechanism for experimentation and risk-taking. A feature that is considered unstable or risky must not be incubated in a JDK feature release. Rather, the feature must be iterated and stabilized in its own project, which in turn produces binaries that are clearly distinguished from binaries of the JDK Project. Incubation is intended to offer a glance at the summit, not the route to base camp.

This JEP does not constrain the shape of an incubating feature. An incubating language feature may add declaration, statement, or expression forms to the syntax of the Java language; it may modify the static semantics (typing) of pre-existing declarations, statements, or expressions; and it may modify the dynamic semantics (execution) of statements or expressions. An incubating VM feature may make additions and modifications of a similar nature to the class file format and the bytecode instruction set. Unlike incubating APIs, which have their own jdk.incubator namespace for packages and modules, there is no requirement that an incubating language or VM feature adopts a striking syntactic form to denote that the feature is incubating.

As a potential scenario for incubation, consider the changing role of the _ character. It was redefined from an identifier to a keyword in order to reserve it for future language features, but this occurred over the course of two JDK releases in order to provide a migration period for existing code. Specifically, the use of _ as an identifier caused javac to give a warning in JDK 8 and an error in JDK 9. If incubation had been available, Java SE 8 could have specified an incubating language feature that defined _ as a keyword. Thus, the use of _ as an identifier would cause javac in JDK 8 to give a warning unless incubation was enabled, in which case it would give an error, presaging the behavior of JDK 9. In Java SE 9, the definition of _ as a keyword would become a final and permanent feature, so javac in JDK 9 would always give an error for the use of _ as an identifier.

Use of Incubating Features

Because incubating language and VM features have not achieved final and permanent status in the Java SE Platform, they are unavailable by default at compile time and run time. Developers who wish to use incubating language features in their programs must explicitly enable them in the compiler and the runtime system. That is, developers must "opt in" twice: once at compile time, when Java source code uses incubating language features, and again at run time, when the corresponding class files are executed. These class files are special; they should not be distributed beyond the control of the developer who chose to use incubating language features.

In order to allow the runtime system to detect when "opt in" is necessary, compilers must emit class files which (i) memorialize the use of incubating language features by the original source code, and (ii) record the use of incubating VM features in the constant pool entries and attributes of the class files. In more detail:

A class file denotes that it depends on the incubating VM features of Java SE $N by having a major_version item that corresponds to Java SE $N and a minor_version item that has all 16 bits set. (Using minor_version is preferred to access_flags, which already very busy insofar as every bit is already allocated to a flag if viewing the union of access_flags in ClassFile+field_info+method_info; also, it would be best to leave unused bits for actual incubating features. Using minor_version is preferred to a custom attribute or a special constant pool entry because the VM's processing of the constant pool may depend on whether incubation is enabled, and those constructs would be read too late.) The unsubtle bit pattern in minor_version is deliberately chosen in preference to an individual bit. (We wish to avoid well-meaning but ultimately misguided guesses at future roles, and therefore bit patterns, for the item. We also wish to avoid the particular confusion around the lowest bit: if a Java SE 11 class file with incubating content had version 55.1, it would be easy to interpret incorrectly as the "next" version after 55.0.)

A JVM implementation for Java SE $N must not define a class file that depends on the incubating VM features of another platform version, even if the JVM implementation would otherwise understand the class file's version. Specifically, ClassLoader.defineClass must fail for the bytes of the class file. This prevents the class files from running years into the future, long after the incubating language features that were enjoyed by the developer have been dropped or finalized in a different form. In essence, Java SE $N+1 does not claim backwards compatibility with the incubating features of Java SE $N.

If incubating language features are not enabled at compile time, developers should not expect the compiler to compile source code which uses incubating language features of any Java SE version, or to compile source code against class files which depend on incubating VM features of any Java SE version. Similarly, if incubating VM features are not enabled at run time, a JVM implementation will not load a class file that depends on the incubating VM features of any Java SE version.

Example use of incubating features

This JEP mandates the presence of a mechanism to enable incubating features, but does not specify that mechanism. To aid understanding, this section outlines a possible mechanism to enable incubating features in the command-line tools of the JDK.

The mechanism is a single command-line flag, --incubating. For simplicity, the same flag is used by all JDK tools. For clarity, the flag takes a version number denoting the release of the Java SE Platform which defined the incubating features. On JDK $N, the only acceptable version number is $N itself; there is no expectation that JDK $N will understand incubating features from earlier releases.

(This specificity is deliberate. Developers must be both precise and accurate when enabling incubating features. The tools demand a precise version number to ensure that developers who rely on the incubating features of JDK $N don't accidentally get different incubating features when migrating to JDK $N+1. Without a version number, the meaning of the flag would change silently from one JDK to the next. The tools also demand an accurate version number -- the version of the current JDK and not any prior JDK -- to allow for the fact that incubating features from prior JDKs may not survive into subsequent JDKs. It would be costly for, say, JDK 12 to support legacy incubating features from JDK 11 which were explicitly dropped for Java SE 12. Finally, tools should recognize that each incubating feature is just as much a part of the Java SE Platform as every other incubating feature, so it would be inappropriate to allow incubating features to be enabled on an individual basis.)

Compile time

javac takes a flag to enable the incubating language features of the current JDK feature release.

On JDK 11:

javac Foo.java                  // Do not enable any incubating features
javac --incubating 11 Foo.java  // Enable incubating features of Java SE 11

On JDK 12:

javac Foo.java                  // Do not enable any incubating features
javac --incubating 12 Foo.java  // Enable incubating features of Java SE 12

ALLOWED:
javac --release 12 --incubating 12 Foo.java
DISALLOWED:
javac --release 12 --incubating 11 Foo.java

ALLOWED:
javac --release 11 Foo.java
DISALLOWED:
javac --release 11 --incubating 11 Foo.java
javac --release 11 --incubating 12 Foo.java

When the incubating language features of Java SE $N are enabled (--incubating $N):

Whether incubating language features are enabled or not, javac in JDK $N prints a message if it detects the use of an incubating language feature of Java SE $N in source code, akin to the message for use of a deprecated API:

Note: Some input files use an incubating language feature.
Note: Recompile with -Xlint:incubation for details.

The off-by-default "incubation" analysis provides information about each use of an incubating language feature of Java SE $N. Additionally, on a best effort basis, the analysis may provide information about the use of incubating language features of other Java SE versions.

jshell and javadoc in JDK $N support the incubating language features of Java SE $N only if --incubating $N is specified at startup.

Run time

The java launcher takes a flag to enable the incubating VM features of the current JDK feature release. This enables execution of any class file that depends on those incubating VM features, either because it truly refers to those features or because it was compiled from source code which used incubating language features.

On JDK 11:

java Foo                           // Do not enable any incubating features
java --incubating 11 Foo           // Enable incubating features of SE 11
java --incubating 11 -jar App.jar  // Enable incubating features of SE 11
java --incubating 11 -m App        // Enable incubating features of SE 11

On JDK 12:

java Foo                           // Do not enable any incubating features
java --incubating 12 Foo           // Enable incubating features of SE 12
java --incubating 12 -jar App.jar  // Enable incubating features of SE 12
java --incubating 12 -m App        // Enable incubating features of SE 12

DISALLOWED:
java --incubating 11 Foo

When the incubating VM features of Java SE $N are enabled (--incubating $N):

Whether incubating VM features are enabled or not, class files that do not depend on incubating VM features are loaded, linked, and executed in the ordinary way.

javap in JDK $N does not offer a --incubating flag. It automatically highlights where a class file depends on incubating VM features of Java SE $N. For a class file that depends on incubating VM features of an earlier Java SE version, javap in JDK $N makes a best effort to render the incubating content.

Relationship to Incubating APIs

An incubating API is an API of non-trivial size that is under development for eventual inclusion in the Java SE Platform or the JDK, but is not yet sufficiently proven. There is no connection between the evaluation and finalization processes used for incubating APIs, and the analogous processes used for incubating language and VM features.

An incubating language or VM feature must not rely on an incubating API, since the API is not part of the Java SE Platform. For example, if an incubating language feature was to expand the throw and catch statements to support a third kind of exception class (alongside the checked exception classes and the unchecked exception classes), then it would not be appropriate for the root class of this new kind (i.e., the analog of java.lang.Exception and java.lang.RuntimeException) to reside in an incubating API.

An incubating language or VM feature may be associated informally with an incubating API that is offered for developer convenience. For example, if an incubating language feature introduced multi-line strings, then it may be associated with new string-processing methods (e.g., to strip newlines, adjust indentation, etc) which are exposed via a class in the package jdk.incubator.strings long before they are committed to java.lang.String.

The implementation of an incubating language or VM feature may rely on incubating APIs. For example, if javac implements multi-line strings (an incubating language feature) by storing them in a new kind of constant pool entry (an incubating VM feature), then javac may emit class files whose bytecode retrieves multi-line strings by invoking methods in the package jdk.incubator.strings.classfile (an incubating API). As a convenience for developers, the incubator module which contains this package would be resolved automatically when the --incubating flag is passed to javac or the java launcher. Finally, it is conceivable that the implementation of an incubating language or VM feature could rely on an incubating API whose own implementation relies on a different incubating language or VM feature; implementers may have to collaborate in order to avoid circularity.

Process issues

In the JEP Process, every new language and VM feature is drafted as a JEP and moves through various phases of candidacy and endorsement before being integrated into a JDK feature release. It is envisaged that a JEP can be drafted, reviewed, endorsed, submitted, and have development work start without regard to whether the feature will be integrated as an incubating feature. This affirms, for a given release of the Java SE Platform, that the technical quality of a incubating feature is equal to that of a permanent feature.

In other words, incubation should not be top of mind when a feature is designed and developed. Still, at some point, a JEP owner may warm to the idea of incubation. The owner is free to reflect the possibility of incubation in the feature's implementation (such as the choice of package names for internal classes) and in related processes (such as compatibility review). However, the owner should be measured about reflecting incubation in the JEP itself. For example, it would be inappropriate for the JEP's title to mention incubation, although it would be acceptable for the JEP to discuss how incubation could benefit the feature. It would also be acceptable for the JEP to consider the possibility of the feature being incubated and later removed. For example, the owner could mitigate the cost of removal by evaluating the difficulty of dropping the implementation from the JDK.

The formal step to incubation occurs when the JEP reaches Proposed To Target status. At that time, the JEP owner must state if they wish the feature to incubate in the proposed release. Subsequently, if the JEP reaches Targeted status, it will be targeted as either incubating or permanent. After the JEP is Targeted, it cannot easily change from incubating to permanent, or vice versa; it must return to Candidate status first.

After the (incubating) feature has been integrated into the Reference Implementation of the Java SE Platform, the JEP is closed as usual. A considerable amount of evangelism lies ahead for the JEP owner! The technique or channel by which the owner requests or receives feedback from developers is left to the discretion and experience of the owner. Similarly, the criteria for deciding whether an incubating feature of Java SE $N should become permanent in Java SE $N+1 (or, less commonly, should re-incubate in Java SE $N+1) is left to the judgment and expertise of the Specification Lead of the Umbrella JSR for the Java SE $N+1 Platform. If the decision is to remove the incubating feature, then the owner should file an issue in JBS to remove the feature in JDK $N+1; no new JEP is needed. On the other hand, if the decision is to finalize or re-incubate, then the owner should file a new JEP, noting refinements informed by developer feedback; this JEP will ultimately reach Targeted status for a future Java SE Platform.

Alternatives

The JDK Project could publish Early Access (EA) binaries during stabilization of the release, prior to General Availability (GA). EA binaries may give new features the broad exposure necessary to provoke useful feedback. If refinements could be designed, specified, and implemented prior to GA, then there would be no need to incubate the features in a GA release. Historically, however, EA binaries have not been widely downloaded and tested by developers at large; this is likely to be even more true when GA releases occur every six months.

In some cases, project-specific EA binaries are an appropriate alternative carrier for features that need feedback:

A different way to include disabled features in a GA release is to hide them behind existing flags for conditional behavior. These include -Xlint:future in javac and -Xfuture in HotSpot. Alternatively, source code which uses an incubating language feature could have an annotation on the class or method declaration to indicate that javac should enable the incubating feature automatically. (One benefit of this local, source-driven scheme is that the annotation could denote which Java SE Platform version the incubating feature is from. Later versions of javac would refuse to compile the code, avoiding the problem where the feature's semantics, but not syntax, changed after incubation and would thus "silently" modify the meaning of the code.)

Risks and Assumptions

Under the six-month cadence, a language or VM feature which "misses the train" has only a short wait before the chance to catch the next one. The decision to incubate a language or VM feature means including the feature in the stabilization branch, so the train has been caught, yet the final feature is not truly on board. This could cause confusion. Worse, it might tempt a feature owner who is in danger of missing the train to label their feature as incubating in order to catch it.

It is assumed that the time frame for feedback on an incubating language or VM feature (typically six months, but more if it re-incubates) is sufficient to allow non-trivial course corrections. However, because exploiting new language features can demand substantial refactoring of existing programs, the time frame may not be sufficient for thorough exploration by developers.