JEP 220: Modular Run-Time Images

AuthorMark Reinhold
OwnerAlan Bateman
Created2014/10/23 15:05
Updated2016/06/30 20:01
TypeFeature
StatusIntegrated
ScopeSE
Discussionjigsaw dash dev at openjdk dot java dot net
EffortL
DurationL
Priority1
Reviewed byAlan Bateman, Alex Buckley, Chris Hegarty, Mandy Chung, Paul Sandoz
Release9
Issue8061971
BlocksJEP 200: The Modular JDK
JEP 261: Module System
Relates toJEP 162: Prepare for Modularization
JEP 201: Modular Source Code
JEP 282: jlink: The Java Linker

Summary

Restructure the JDK and JRE run-time images to accommodate modules and to improve performance, security, and maintainability. Define a new URI scheme for naming the modules, classes, and resources stored in a run-time image without revealing the internal structure or format of the image. Revise existing specifications as required to accommodate these changes.

Goals

Success Metrics

Modular run-time images equivalent to the JRE, JDK, and Compact Profile images of the immediately-preceding JDK 9 build must not regress on a representative set of startup, static footprint, and dynamic footprint benchmarks.

Non-Goals

Motivation

Project Jigsaw aims to design and implement a standard module system for the Java SE Platform and to apply that system to the Platform itself, and to the JDK. Its primary goals are to make implementations of the Platform more easily scalable down to small devices, improve the security and maintainability, enable improved application performance, and provide developers with better tools for programming in the large.

This JEP is the third of four JEPs planned for Project Jigsaw. The earlier JEP 200 defines the structure of the modular JDK, and JEP 201 reorganized the JDK source code into modules. A later JEP will introduce the actual module system.

Description

Current run-time image structure

The JDK build system presently produces two types of run-time images: A Java Runtime Environment (JRE), which is a complete implementation of the Java SE Platform, and a Java Development Kit (JDK), which embeds a JRE and includes development tools and libraries. (The three Compact Profile builds are subsets of the JRE.)

The root directory of a JRE image contains two directories, bin and lib, with the following content:

A JDK image includes a copy of the JRE in its jre subdirectory and contains additional subdirectories:

The root directory of a JDK image, or of a JRE image that is not embedded in a JDK image, also contains various COPYRIGHT, LICENSE and README files and also a release file that describes the image in terms of simple key/value property pairs, e.g.,

JAVA_VERSION="1.9.0"
OS_NAME="Linux"
OS_VERSION="2.6"
OS_ARCH="amd64"

New run-time image structure

The present distinction between JRE and JDK images is purely historical, a consequence of an implementation decision made late in the development of the JDK 1.2 release and never revisited. The new image structure will eliminate this distinction: A JDK image will simply be a run-time image that happens to contain the full set of development tools and other items historically found in the JDK.

A modular run-time image will contain the following directories:

The root directory of a modular run-time image will also, of course, contain the necessary COPYRIGHT, LICENSE, README, and release files. To make it easy to tell which modules are present in a run-time image the release file will be augmented with a new property, MODULES, which will be a space-separated list of the names of those modules. The list will be topologically ordered according to the modules' dependence relationships, so the java.base module will always be first.

Removed: The endorsed-standards override mechanism

The endorsed-standards override mechanism allows implementations of newer versions of standards maintained outside of the Java Community Process, or of standalone APIs that are part of the Java SE Platform yet continue to evolve independently, to be installed into a run-time image.

The endorsed-standards mechanism is presently defined in terms of a path-like system property, java.endorsed.dirs, and a default value for that property, $JAVA_HOME/lib/endorsed. A jar file containing a newer implementation of an endorsed standard or standalone API can be installed into a run-time image by placing it in one of the directories named by the system property, or by placing it in the default lib/endorsed directory if the system property is not defined. Such jar files are prepended to the JVM's bootstrap class path at run time, thereby overriding any definitions stored in the run-time system itself.

A modular image is composed of modules rather than jar files. Going forward we expect to support endorsed standards and standalone APIs in modular form only, via the concept of upgradeable modules. We therefore propose to remove the java.endorsed.dirs system property, the lib/endorsed directory, and the code that implements this mechanism. To help identify any existing uses of this mechanism we will modify the compiler and the launcher to fail if this system property is set or if the lib/endorsed directory exists.

Removed: The extension mechanism

The extension mechanism allows jar files containing APIs that extend the Java SE Platform to be installed into a run-time image so that their contents are visible to every application that is compiled with or runs on that image.

The mechanism is defined in terms of a path-like system property, java.ext.dirs, and a default value for that property composed of $JAVA_HOME/lib/ext and a platform-specific system-wide directory (e.g, /usr/java/packages/lib/ext on Linux). It works in much the same manner as the endorsed-standards mechanism except that jar files placed in an extension directory are loaded by the run-time environment's extension class loader, which is a child of the bootstrap class loader and the parent of the system class loader, which actually loads the application to be run from the class path. Extension classes therefore cannot override the JDK classes loaded by the bootstrap loader but they are loaded in preference to classes defined by the system loader and its descendants.

The extension mechanism was introduced in JDK 1.2, which was released in 1998, but in modern times we have seen little evidence of its use. This is not surprising, since most Java applications today place the libraries that they need directly on the class path rather than require that those libraries be installed as extensions of the run-time system.

It is technically possible, though awkward, to continue to support the extension mechanism in the modular JDK. To simplify both the Java SE Platform and the JDK we propose to remove the java.ext.dirs system property, the lib/ext directory, and the code that implements this mechanism. To help identify any existing uses of this mechanism we will modify the compiler and the launcher to fail if this system property is set or if the lib/ext directory exists. The compiler and the launcher will ignore the platform-specific system-wide extension directory by default, but if the -XX:+CheckEndorsedAndExtDirs command-line option is specified then they will fail if that directory exists and is not empty.

Several features associated with the extension mechanism will be retained, since they are useful in their own right:

The extension class loader will be retained in order to maintain compatibility. For a class Foo loaded by the system class loader, in particular, the expression

Foo.class.getClassLoader().getParent() != null

will remain true.

Removed: rt.jar and tools.jar

The class and resource files previously stored in lib/rt.jar, lib/tools.jar, lib/dt.jar, and various other internal jar files will now be stored in a more efficient format in implementation-specific files in the lib directory. The format of these files will not be specified and is subject to change without notice.

The removal of rt.jar and similar files leads to three distinct problems:

  1. Existing standard APIs such as the ClassLoader::getSystemResource method return URL objects to name class and resource files inside the run-time image. For example, when run on JDK 8 the code

    ClassLoader.getSystemResource("java/lang/Class.class");

    returns a jar URL of the form

    jar:file:/usr/local/jdk8/jre/lib/rt.jar!/java/lang/Class.class

    which, as can be seen, embeds a file URL to name the actual jar file within the run-time image. The getContent method of that URL object can be used to retrieve the content of the class file, via the built-in protocol handler for the jar URL scheme.

    A modular image will not contain any jar files, so URLs of the above form make no sense. The specifications of getSystemResource and related methods, fortunately, do not require the URL objects returned by these methods actually to use the jar scheme. They do, however, require that it be possible to load the content of a stored class or resource file via these URL objects.

  2. The java.security.CodeSource API and security-policy files use URLs to name the locations of code bases that are to be granted specified permissions. Components of the run-time system that require specific permissions are currently identified in the lib/security/java.policy file via file URLs. The elliptic-curve cryptography provider, e.g., is identified as

    file:${java.home}/lib/ext/sunec.jar

    which, obviously, will have no meaning in a modular image.

  3. IDEs and other kinds of development tools require the ability to enumerate the class and resource files stored in a run-time image, and to read their contents. Today they often do this directly by opening and reading rt.jar and similar files. This will, of course, not be possible with a modular image.

New URI scheme for naming stored modules, classes, and resources

To address the above three problems we propose to define a new URL scheme, jrt, for naming the modules, classes, and resources stored in a run-time image without revealing the internal structure or format of the image.

A jrt URL is a hierarchical URI, per RFC 3986, with the syntax

jrt:/[$MODULE[/$PATH]]

where $MODULE is an optional module name and $PATH, if present, is the path to a specific class or resource file within that module. The meaning of a jrt URL depends upon its structure:

These three forms of jrt URLs address the above problems as follows:

  1. APIs that presently return jar URLs will now return jrt URLs. The above invocation of ClassLoader::getSystemResource, e.g., will now return the URL

    jrt:/java.base/java/lang/Class.class

    A built-in protocol handler for the jrt scheme will be defined so that the getContent method of such URL objects retrieves the content of the named class or resource file.

  2. Security-policy files and other uses of the CodeSource API can use jrt URLs to name specific modules for the purpose of granting permissions. The elliptic-curve cryptography provider, e.g., can now be identified by the jrt URL

    jrt:/jdk.crypto.ec

    Other modules that are currently granted all permissions but do not actually require them can trivially be de-privileged, i.e., given precisely the permissions they require.

  3. A built-in NIO FileSystem provider will be defined for the jrt URL scheme so that development tools can enumerate and read the class and resource files in a run-time image by loading the FileSystem named by the URL jrt:/, as follows:

    FileSystem fs = FileSystems.getFileSystem(URI.create("jrt:/"));
    byte[] jlo = Files.readAllBytes(fs.getPath("java.base",
                                               "java/lang/Object.class"));

    For tools that support the development of code for JDK 9 but which themselves run on JDK 8, a copy of this filesystem provider suitable for use on JDK 8 will be placed in the root directory of JDK 9 run-time images, in a file named jrt-fs.jar.

(The jrt URL protocol handler will not return any content for URLs of the second and third forms.)

Build-system changes

The build system will be modified to produce the new run-time image format described above. We will also take the opportunity here, finally, to rename the images/j2sdk-image, images/j2re-image, and images/j2re-compact$N-image directories to images/jdk, images/jre, and images/jre-compact$N, respectively.

Minor specification changes

JEP 162, implemented in JDK 8, made a number of changes to prepare the Java SE Platform and the JDK for the modularization work proposed here and in related JEPs. Among those changes were the removal of normative specification statements that require certain configuration files to be looked up in the lib directory of run-time images, since those files will now be placed in the conf directory. Most of the SE-only APIs with such statements were so revised as part of Java SE 8, but some APIs shared across the Java SE and EE Platforms still contain such statements:

These statements will be revised so as not to mandate the lib directory as they do now.

Open issues

  1. Some changes to how fonts are configured may yet be required.

  2. The lib/security directory still contains two jar files, whose contents are simply the local and US-export cryptography policy files; we intend to replace these jar files with their content (8061842).

  3. The lib/$ARCH directory is only present in Linux and Solaris builds. It is a vestigial remnant of images that could support multiple CPU architectures, which is no longer a requirement. We will investigate whether its content can be placed directly in the lib directory, as is done on Mac OS and Windows, in which case the lib/$ARCH directory will no longer be needed (8066474).

  4. The content of the demo, sample, and man directories should ideally be derived from appropriate modules; we will investigate how best to do this (8066476).

  5. The javax.activation.MailcapCommandMap class, and related classes, specify ${java.home}/lib/mailmap (JSR 925); this needs to be revised so as not to mandate the lib directory.

Testing

Some existing tests make direct use of run-time image internals (e.g., rt.jar) or refer to system properties (e.g., java.ext.dirs) that no longer exist. These tests will be fixed.

We plan to publish early-access builds containing these changes and then encourage members of the wider Java community to test their tools, libraries, and applications against these builds to help tease out any remaining compatibility issues.

Risks and Assumptions

The central risks of this proposal are ones of compatibility, summarized as follows:

It is impossible to determine the full impact of these changes in the abstract. We must therefore rely upon extensive internal and—especially—external testing. Sophisticated applications such as IDEs are more likely to be affected by these changes than are straightforward libraries and simpler applications. If some of these changes prove to be insurmountable hurdles for developers, deployers, or end users then we will investigate ways to mitigate their impact.

Dependences

This JEP is the third of four JEPs for Project Jigsaw. It depends upon JEP 201, which reorganized the JDK source code into modules and upgraded the build system to compile modules. It also depends upon earlier preparatory work done in JEP 162, implemented in JDK 8.