JEP 12: Preview Language and VM Features

OwnerAlex Buckley
Created2018/01/19 01:27
Updated2018/05/03 21:40
TypeInformational
StatusActive
ScopeSE
Discussionjdk dash dev at openjdk dot java dot net
EffortM
DurationM
Priority2
Reviewed byAlan Bateman, Brian Goetz, Mark Reinhold
Endorsed byMark Reinhold
Issue8195734

Summary

A preview language or VM feature is a new feature of the Java SE Platform that is fully specified, fully implemented, and yet impermanent. It is available in a JDK feature release to provoke developer feedback based on real world use; this may lead to it becoming permanent in a future Java SE Platform.

Goals

Non-Goals

Motivation

The Java SE Platform has global reach, so the cost of a mistake in the design of a Java language or VM feature is high. ("VM feature" encompasses the class file format and the JVM bytecode instruction set.) A mistake may be a hard technical error (such as a flaw in the type system), a soft usability problem (such as a surprising interaction with an older feature), or a poor architectural choice (such as one that forecloses on directions for future features).

To build confidence in the correctness and completeness of a new language or VM feature, it is desirable for the feature to enjoy a period of broad exposure after its specification and implementation are stable but before it achieves final and permanent status in the Java SE Platform. To achieve the broadest possible exposure, and to maximize the likelihood of swift feedback, the feature may be included in a mainline JDK feature release on a preview basis. Previewing a feature in the JDK will also encourage tool vendors to build good support for the feature before the bulk of Java developers use it in production.

Before the next JDK feature release, the feature's "real world" strengths and weaknesses will be evaluated to decide if the feature has a long-term role in the Java SE Platform and, if so, whether it needs refinement. Consequently, the feature may be granted final and permanent status (with or without refinements), or undergo a further preview period (with or without refinements), or else be removed.

Description

Design of a Preview Feature

A preview language or VM feature is a new feature whose design, specification, and implementation are all complete, but which would benefit from a period of broad exposure and evaluation before either achieving final and permanent status in the Java SE Platform or else being refined or removed.

The key properties of a preview language or VM feature are:

  1. High quality. A preview feature must display the same level of technical excellence and finesse as a final and permanent feature of the Java SE Platform. For example, a preview feature must respect traditional Java principles such as readability and compatibility, and it must receive appropriate treatment in the reflective and debugging APIs of the Java SE Platform. As a general rule, for anything not explicitly spelled out in this JEP, the policy is "No different for a preview feature than for a final and permanent feature."

  2. Not experimental. A language or VM feature that is considered experimental, risky, incomplete, or unstable must not be previewed in a JDK feature release. An experimental feature must be iterated and stabilized in its own project, which in turn produces binaries that are clearly distinguished from binaries of the JDK Project. For the purpose of comparison, if an experimental feature is considered 25% "done", then a preview feature should be at least 95% "done". To make a further comparison, the level of completeness and stability expected for a preview language or VM feature is considerably higher than the level expected for an incubating API.

  3. Universally available. The Umbrella JSR for the Java SE $N Platform enumerates the preview features in the platform. The desire for feedback and the expectation of quality means that these features are not "optional"; they must all be supported in full by every implementation of the Umbrella JSR. They are specified in appendices of the Java SE $N Editions of the Java Language Specification (JLS) and the Java Virtual Machine Specification (JVMS). The Java Compatibility Kit for Java SE $N checks conformance of an implementation's preview features to the appropriate appendix.

This JEP does not constrain the shape of a preview feature. A preview language feature may add declaration, statement, expression, and literal forms to the syntax of the Java language; it may modify the static semantics (typing) of pre-existing declarations, statements, expressions, and literals; and it may modify the dynamic semantics (execution) of statements or expressions. A preview VM feature may make additions and modifications of a similar nature to the class file format and the bytecode instruction set. Unlike incubating APIs, which have their own jdk.incubator namespace for packages and modules, there is no requirement that a preview language or VM feature adopts a striking syntactic form to denote that the feature is available on a preview basis.

As a potential example of a preview feature, consider the changing role of the _ character. It was redefined from an identifier to a keyword in order to reserve it for future language features. To provide a migration period for existing code, the redefinition was spread over two JDK releases: in JDK 8, the use of _ as an identifier caused javac to give a warning, while in JDK 9, it caused an error. In the regime described by this JEP, Java SE 8 would specify a preview language feature that defines _ as a keyword. javac in JDK 8 would give a warning for the use of _ as an identifier, unless preview features are enabled, in which case javac would give an error. Subsequently, Java SE 9 would specify the definition of _ as a keyword on a final and permanent basis, so javac in JDK 9 would always give an error for the use of _ as an identifier, even when preview features are not enabled.

Use of Preview Features

Because preview language and VM features have not achieved final and permanent status in the Java SE Platform, they are unavailable by default at compile time and run time. Developers who wish to use preview language features in their programs must explicitly enable them in the compiler and the runtime system. That is, developers must "opt in" twice: once at compile time, when Java source code uses preview language features, and again at run time, when the corresponding class files are executed. These class files are special; they should not be distributed beyond the control of the developer who chose to use preview language features.

In order to allow the runtime system to detect when "opt in" is necessary, compilers must emit class files which (i) memorialize the use of preview language features by the original source code, and (ii) record the use of preview VM features in the constant pool entries and attributes of the class files. In more detail:

A class file denotes that it depends on the preview VM features of Java SE $N by having a major_version item that corresponds to Java SE $N and a minor_version item that has all 16 bits set. For example, a class file that depends on the preview VM features of Java SE 11 would have version 55.65535.

Using minor_version is preferred to access_flags, which is already very busy insofar as every bit is allocated to a flag when viewing the union of access_flags across the ClassFile, field_info, and method_info structures. Unused bits in any one structure are best left for use by actual features.

Using minor_version is also preferred to a custom attribute or a special constant pool entry because the VM's processing of the constant pool may depend on whether preview features are enabled, and those constructs would be read too late.

The unsubtle bit pattern in minor_version is deliberately chosen in preference to an individual bit. We wish to avoid well-meaning but ultimately misguided guesses at future roles, and therefore bit patterns, for the item. We also wish to avoid the particular confusion around the lowest bit: if a Java SE 11 class file with preview content had version 55.1, it would be easy to interpret incorrectly as the "next" version after 55.0.

A JVM implementation for Java SE $N must not define a class file that depends on the preview VM features of another platform version, even if the JVM implementation would otherwise understand the class file's version. Specifically, ClassLoader.defineClass must fail for the bytes of the class file. This prevents the class files from running years into the future, long after the preview language features that were enjoyed by the developer have been removed or finalized in a different form. In essence, Java SE $N+1 does not claim backwards compatibility with the preview features of Java SE $N.

If preview language features are not enabled at compile time, developers should not expect the compiler to compile source code which uses preview language features of any Java SE version, or to compile source code against class files which depend on preview VM features of any Java SE version. Similarly, if preview VM features are not enabled at run time, a JVM implementation will not load a class file that depends on the preview VM features of any Java SE version.

Example use of preview features

This JEP mandates the presence of a mechanism to enable preview features, which are otherwise disabled. This JEP also forbids the mechanism from enabling preview features on an individual basis, because all preview features have equal status in the Java SE Platform.

This JEP does not, however, specify an actual mechanism. Therefore, to aid understanding, this section outlines a possible mechanism for enabling preview features in the command-line tools of the JDK: javac, java, javadoc, jshell, and possibly jlink. The mechanism is a single command-line flag, --enable-preview.

Compile time

javac in JDK $N accepts the --enable-preview flag only in combination with --release $N or -source $N. (For brevity, only --release is mentioned in the rest of this JEP.)

The meaning of --enable-preview changes from one JDK to the next. Requiring the developer to spell out a concrete version number with --release sets the expectation that code relies on the preview features found specifically in JDK $N, and may not compile in future if JDK $N+1 has different preview features.

--enable-preview itself does not take a version number because it would be easy to misinterpret. For a developer using JDK 11, the flag --enable-preview 12 appears to suggest a commitment of what JDK 12 will contain, but no such commitment is possible.

There is no expectation that javac in JDK $N will understand preview features from JDK $N-1 or any other prior release. For example, it would be costly for JDK 12 to support preview features from JDK 11 which were changed or dropped in response to feedback. Consequently, --enable-preview is illegal if the operand to --release is not $N.

On JDK 11:

javac Foo.java                                // Do not enable any preview features
javac --release 11 --enable-preview Foo.java  // Enable all preview features of Java SE 11
javac --release 10 --enable-preview Foo.java  // DISALLOWED

On JDK 12:

javac Foo.java                                // Do not enable any preview features
javac --release 12 --enable-preview Foo.java  // Enable all preview features of Java SE 12
javac --release 11 --enable-preview Foo.java  // DISALLOWED

When the preview language features of Java SE $N are enabled (--release $N --enable-preview):

Whether preview language features are enabled or not, javac in JDK $N prints a message if it detects the use of a preview language feature of Java SE $N in source code, akin to the message for use of a deprecated API:

Note: Some input files use a preview language feature.
Note: Recompile with -Xlint:preview for details.

The off-by-default "preview" analysis suggested by this message provides information about each use of a preview language feature of Java SE $N. Additionally, on a best effort basis, the analysis may provide information about the use of preview language features of other Java SE versions.

jshell and javadoc in JDK $N support the preview language features of Java SE $N only if --enable-preview is specified at startup.

Run time

The java launcher in JDK $N takes a flag to enable the preview VM features of Java SE $N. This enables execution of any class file that depends on those preview VM features, either because it truly refers to those features or because it was compiled from source code which used preview language features.

On JDK 11:

java Foo                            // Do not enable any preview features
java --enable-preview Foo           // Enable all preview features of SE 11
java --enable-preview -jar App.jar  // Enable all preview features of SE 11
java --enable-preview -m App        // Enable all preview features of SE 11

On JDK 12:

java Foo                            // Do not enable any preview features
java --enable-preview Foo           // Enable all preview features of SE 12
java --enable-preview -jar App.jar  // Enable all preview features of SE 12
java --enable-preview -m App        // Enable all preview features of SE 12

When the preview VM features of Java SE $N are enabled (--enable-preview):

Whether preview VM features are enabled or not, class files that do not depend on preview VM features are loaded, linked, and executed in the ordinary way.

javap in JDK $N does not accept the --enable-preview flag. It automatically highlights where a class file depends on preview VM features of Java SE $N. For a class file that depends on preview VM features of an earlier Java SE version, javap in JDK $N makes a best effort to render the preview content.

Relationship to Java SE APIs

A preview language or VM feature in Java SE $N may rely on any API that is final and permanent in Java SE $N-1. For example, if multi-catch had been a preview feature, then it could legitimately have relied on the longstanding hierarchy of exception types under java.lang.Throwable.

More interestingly, a preview language or VM feature will often be co-developed with new java.* APIs. Such APIs fall into three buckets:

  1. Essential. The API exists because code cannot enjoy the preview feature without it. For example, the enhanced-for statement relies on java.lang.Iterable, and the try-with-resources statement relies on java.lang.AutoCloseable. The JLS will refer to the API in normative text.

  2. Reflective. The API exists to expose the preview feature in the Core Reflection API, Method Handle API, Language Model API, Annotation Processing API, Compiler API, or JNI. Typically, this exposure is only necessary if the preview feature is a declaration form (such as enum types or record types) rather than a statement, expression, or literal form. For example, repeatable annotation types prompted additional methods in java.lang.Class. The JLS will not refer to the API in normative text, but may refer to it in non-normative text.

  3. Convenient. The API is a collection of useful types and methods that promote or assist usage of the preview feature, but are not essential for it. For example, the Streams API is convenient for developers, especially when lambda expressions are given as arguments, but lambda expressions do not technically rely on it. The JLS is unlikely to refer to the API in any way.

If a preview language or VM feature does not eventually achieve final and permanent status in the Java SE Platform, then its essential and reflective APIs must be removed swiftly. It is important to communicate this possibility when they are introduced; after all, they are Java SE APIs. This JEP does not propose a new mechanism for flagging "preview APIs", but rather, proposes to reuse the deprecation mechanism:

  1. Essential and reflective APIs that are connected with a preview feature should be terminally deprecated at birth, that is, annotated @Deprecated(forRemoval=true, since=...) when they are introduced.

  2. The narrative specifications of essential and reflective APIs should indicate that their existence is tied to the success of a preview feature -- "Use at your own risk."

  3. The narrative specification of an essential API should discourage its use for purposes that go beyond the preview feature. For example, if enhanced-for was a preview feature, then its java.lang.Iterable interface should not be used as a general-purpose abstraction in method signatures; the convenient availability of the interface may be a mirage.

  4. When a Java compiler gives a removal warning for a use of a terminally deprecated API connected with a preview feature, it is recommended that the text of the warning be customized to indicate the connection. (This is a quality of implementation issue; the generation of a warning is mandated, as is the ability to suppress it in source code, but the precise text is not.)

If a preview language or VM feature does achieve final and permanent status in the Java SE Platform, then its essential and reflective APIs will be undeprecated.

Relationship to Incubating APIs

An incubating API is an API of non-trivial size that is under development for eventual inclusion in the Java SE Platform or the JDK, but is not yet sufficiently proven.

A preview language or VM feature must not rely on an incubating API, since the API is not part of the Java SE Platform. For example, if a preview language feature was to expand the throw and catch statements to support a third kind of exception class (alongside the checked exception classes and the unchecked exception classes), then it would not be appropriate for the root class of this new kind (i.e., the analog of java.lang.Exception and java.lang.RuntimeException) to reside in an incubating API.

A preview language or VM feature may be associated informally with an incubating API that is offered for developer convenience, perhaps as an alternative to the convenient java.* APIs described above. For example, if a preview language feature introduced multi-line string literals, then it may be advertised alongside new string-processing methods (e.g., to strip newlines, adjust indentation, etc) which reside in the jdk.incubator.strings package long before they are committed to the java.lang.String class.

The implementation of a preview language or VM feature may rely on incubating APIs. For example, if javac implements multi-line strings (a preview language feature) by storing them in a new kind of constant pool entry (a preview VM feature), then javac may emit class files whose bytecode retrieves multi-line strings by invoking methods in the package jdk.incubator.strings.classfile (an incubating API). As a convenience to developers, the incubator module which contains this package would be resolved automatically when the --enable-preview flag is passed to javac or the java launcher.

It is conceivable that the implementation of a preview language or VM feature could rely on an incubating API whose own implementation relies on a different preview language or VM feature; implementers may have to collaborate in order to avoid circularity.

Process issues

In the JEP Process, every new language and VM feature is drafted as a JEP and moves through various phases of candidacy and endorsement before being integrated into a JDK feature release. It is envisaged that a JEP can be drafted, reviewed, endorsed, submitted, and have development work start without regard to whether the feature will be integrated as a preview feature. This affirms, for a given release of the Java SE Platform, that the technical quality of a preview feature is equal to that of a final and permanent feature.

In other words, previewing should not be top of mind when a feature is designed and developed. Still, at some point, a JEP owner may warm to the idea of making the feature available on a preview basis. The owner is free to discuss the implications in the JEP, such as the possibility of the feature being previewed and later removed. (This would be especially appropriate in the somewhat risky situation where the feature relies on new java.* APIs, since user code can refer to them despite strong advice not to. If feedback is likely to cause changes to the APIs, or if they are so attractive that their use will spread far beyond the preview feature, then the JEP owner should consider if previewing is appropriate.) The owner is also free to reflect previewing in the feature's implementation (such as the choice of package names for internal classes) and in related processes (such as compatibility review).

The formal step to preview availability occurs when the JEP reaches Proposed To Target status. At that time, the JEP owner must state if they wish the feature to be available as a preview in the proposed release. If and when the JEP reaches Targeted status, it will be recorded as a preview feature by having "(preview)" appended to its title, e.g., "JEP 326: Raw String Literals (preview)". After the JEP is Targeted, it cannot easily change from preview to permanent, or vice versa; it must return to Candidate status first.

After a preview feature has been integrated into the Reference Implementation of the Java SE Platform, the JEP is closed as usual. A considerable amount of evangelism lies ahead for the JEP owner! The technique or channel by which the owner requests or receives feedback from developers is left to the discretion and experience of the owner. Similarly, the criteria for deciding whether a preview feature of Java SE $N should become permanent in Java SE $N+1 (or, less commonly, should re-preview in Java SE $N+1) is left to the judgment and expertise of the Specification Lead of the Umbrella JSR for the Java SE $N+1 Platform. If the decision is to remove the preview feature, then the owner should file an issue in JBS to remove the feature in JDK $N+1; no new JEP is needed. On the other hand, if the decision is to finalize or re-preview, then the owner should file a new JEP, noting refinements informed by developer feedback; this JEP will ultimately reach Targeted status for a future Java SE Platform.

Alternatives

The JDK Project could publish Early Access (EA) binaries during stabilization of the release, prior to General Availability (GA). EA binaries may give new features the broad exposure necessary to provoke useful feedback. If refinements could be designed, specified, and implemented prior to GA, then there would be no need to preview the features in a GA release. Historically, however, EA binaries have not been widely downloaded and tested by developers at large; this is likely to be even more true when GA releases occur every six months.

In some cases, project-specific EA binaries are a plausible carrier for features that need feedback:

A different way to include disabled features in a GA release is to hide them behind existing flags for conditional behavior. These include -Xlint:future in javac and -Xfuture in HotSpot. Alternatively, source code which uses an preview language feature could have an annotation on the class or method declaration to indicate that javac should enable the preview feature automatically. (One benefit of this local, source-driven scheme is that the annotation could denote which Java SE Platform version the preview feature is from. Later versions of javac would refuse to compile the code, avoiding the problem where the feature's semantics, but not syntax, changed after preview and would thus "silently" modify the meaning of the code.)

Risks and Assumptions

Under the six-month cadence, a language or VM feature which "misses the train" has only a short wait before the chance to catch the next one. The decision to preview a language or VM feature means including the feature in the stabilization branch, so the train has been caught, yet the final and permanent feature is not truly on board. This could cause confusion. Worse, it might tempt a feature owner who is in danger of missing the train to label their feature as a preview in order to catch it.

It is assumed that the time frame for feedback on a preview language or VM feature (typically six months, but more if it re-previews) is sufficient to allow non-trivial course corrections. However, because exploiting new language features can demand substantial refactoring of existing programs, the time frame may not be sufficient for thorough exploration by developers.