JEP 261: Module System

AuthorsAlan Bateman, Alex Buckley, Jonathan Gibbons, Mark Reinhold
OwnerMark Reinhold
Created2014/10/23 15:05
Updated2016/06/22 16:07
TypeFeature
StatusIntegrated
ScopeSE
JSR376
Discussionjigsaw dash dev at openjdk dot java dot net
EffortXL
DurationL
Priority1
Reviewed byAlan Bateman, Alex Buckley, Chris Hegarty, Jonathan Gibbons, Mandy Chung, Paul Sandoz
Endorsed byBrian Goetz
Release9
Issue8061972
BlocksJEP 200: The Modular JDK
JEP 282: jlink: The Java Linker
DependsJEP 220: Modular Run-Time Images
JEP 260: Encapsulate Most Internal APIs

Summary

Implement the Java Platform Module System, as specified by JSR 376, together with related JDK-specific changes and enhancements.

Description

The Java Platform Module System (JSR 376) proposes changes and extensions to the Java programming language, the Java virtual machine, and the standard Java APIs. This JEP will implement that specification. As a consequence, the javac compiler, the HotSpot virtual machine, and the run-time libraries will implement modules as a fundamental new kind of Java program component and provide for the reliable configuration and strong encapsulation of modules in all phases of development.

This JEP will also change, extend, and add JDK-specific tools and APIs, which are outside the scope of the JSR, that are related to compilation, linking, and execution. Related changes to other tools and APIs, e.g., the javadoc tool and the Doclet API, will be the subject of separate JEPs.

This JEP assumes that the reader is familiar with the latest State of the Module System document and also the other Project Jigsaw JEPs:

Phases

To the familiar phases of compile time (the javac command) and run time (the java run-time launcher) we add the notion of link time, an optional phase between the two in which a set of modules can be assembled and optimized into a custom run-time image. The linking tool, jlink, is the subject of JEP 282; many of the new command-line options implemented by javac and java are also implemented by jlink.

Module paths

The javac, jlink, and java commands, as well as several others, now accept options to specify various module paths. A module path is a sequence, each element of which is either a module definition or a directory containing module definitions. Each module definition is either

In the latter case the directory tree can be a compiled module definition, populated with individual class and resource files and a module-info.class file at the root or, at compile time, a source module definition, populated with individual source files and a module-info.java file at the root.

A module path, like other kinds of paths, is specified by a string of path names separated by the host platform's path-separator character (':' on most platforms, ';' on Windows).

Module paths are very different from class paths: Class paths are a means to locate definitions of individual types and resources, whereas module paths are a means to locate definitions of whole modules. Each element of a class path is a container of type and resource definitions, i.e., either a JAR file or an exploded, package-hierarchical directory tree. Each element of a module path, by contrast, is a module definition or a directory which each element in the directory is a module definition, i.e., a container of type and resource definitions, i.e., either a modular JAR file, a JMOD file, or an exploded module directory.

During the resolution process the module system locates a module by searching along several different paths, dependent upon the phase, and also by searching the compiled modules built-in to the environment, in the following order:

The module definitions present on these paths, together with the system modules, define the universe of observable modules.

When searching a module path for a module of a particular name, the module system takes the first definition of a module of that name. Version strings, if present, are ignored; if an element of a module path contains definitions of multiple modules with the same name then resolution fails and the compiler, linker, or virtual machine will report an error and exit. It is the responsibility of build tools and container applications to configure module paths so as to avoid version conflicts; it is not a goal of the module system to address the version-selection problem.

Root modules

The module system constructs a module graph by resolving the transitive closure of the dependences of a set of root modules with respect to the set of observable modules.

When the compiler compiles code in the unnamed module, or the java launcher is invoked and the main class of the application is loaded from the class path into the unnamed module of the application class loader, then the default set of root modules for the unnamed module is computed as follows:

Otherwise, the default set of root modules depends upon the phase:

It is occasionally necessary to add modules to the default root set in order to ensure that specific platform, library, or service-provider modules will be present in the module graph. In any phase the option

-addmods <module>(,<module>)*

where <module> is a module name, adds the named modules to the default set of root modules.

As a special case at run time, if <module> is ALL-DEFAULT then the default set of root modules for the unnamed module, as defined above, is added to the root set. This is useful when the application is a container that hosts other applications which can, in turn, depend upon modules not required by the container itself.

As a further special case at run time, if <module> is ALL-SYSTEM then all system modules are added to the root set, whether or not they are in the default set. This is sometimes needed by test harnesses. This option will cause many modules to be resolved; in general, ALL-DEFAULT should be preferred.

As a final special case, if <module> is ALL-MODULE-PATH then all observable modules found on the relevant module paths are added to the root set. ALL-MODULE-PATH is valid at both compile time and run time. This is provided for use by build tools such as Maven, which already ensure that all modules on the module path are needed. It is also a convenient means to add automatic modules to the root set.

Limiting the observable modules

It is sometimes useful to limit the observable modules for, e.g., debugging, or to reduce the number of modules resolved when the main module is the unnamed module defined by the application class loader for the class path. The -limitmods option can be used, in any phase, to do this. Its syntax is:

-limitmods <module>(,<module>)*

where <module> is a module name. The effect of this option is to limit the observable modules to those in the transitive closure of the named modules plus the main module, if any, plus any further modules specified via the -addmods option.

(The transitive closure computed for the interpretation of the -limitmods option is a temporary result, used only to compute the limited set of observable modules. The resolver will be invoked again in order to compute the actual module graph.)

Increasing readability

When testing and debugging it is sometimes necessary to arrange for one module to read some other module, even though the first module does not depend upon the second via a requires clause in its module declaration. This may be needed, e.g., in order for a module under test to access the test harness itself, or to access libraries related to the harness. The -XaddReads option can be used, at both compile time and run time, to do this. Its syntax is:

-XaddReads:<source-module>=<target-module>

where <source-module> and <target-module> are module names.

The -XaddReads option can be used more than once. The effect of each instance is to add a readability edge from the source module to the target module. This is, essentially, a command-line form of a requires clause in a module declaration, or an invocation of an unrestricted form of the Module::addReads method. As a consequence, code in the source module will be able to access types in packages of the target module so long as each package is exported via an exports clause in the source module's declaration, an invocation of the Module::addExports method, or an instance of the -XaddExports option (defined below).

If, for example, a test harness injects a white-box test class into the java.management module, and that class extends an exported utility class in the (hypothetical) testng module, then the access it requires can be granted via the option

-XaddReads:java.management=testng

As a special case, if the <target-module> is ALL-UNNAMED then readability edges will be added from the source module to all present and future unnamed modules, including that corresponding to the class path. This allows code in modules to be tested by test frameworks that have not, themselves, yet been converted to modular form.

Breaking encapsulation

It is sometimes necessary to violate the access-control boundaries defined by the module system, and enforced by the compiler and virtual machine, in order to allow one module to access some of the unexported types of another module. This may be desirable in order to, e.g., enable white-box testing of internal types, or to expose unsupported internal APIs to code that has come to depend upon them. The -XaddExports option can be used, at both compile time and run time, to do this. Its syntax is:

-XaddExports:<source-module>/<package>=<target-module>(,<target-module>)*

where <source-module> and <target-module> are module names and <package> is the name of a package.

The -XaddExports option can be used more than once, but at most once for any particular combination of source module and package name. The effect of each instance is to add a qualified export of the named package from the source module to the target module. This is, essentially, a command-line form of an exports clause in a module declaration, or an invocation of an unrestricted form of the Module::addExports method. As a consequence, code in the target module will be able to access types in the named package of the source module if the target module reads the source module, either via a requires clause in its module declaration, an invocation of the Module::addReads method, or an instance of the -XaddReads option.

If, for example, the module jmx.wbtest contains a white-box test for the unexported com.sun.jmx.remote.internal package of the java.management module, then the access it requires can be granted via the option

-XaddExports:java.management/com.sun.jmx.remote.internal=jmx.wbtest

As a special case, if the <target-module> is ALL-UNNAMED then the source package will be exported to all unnamed modules, whether they exist initially or are created later on. Thus access to the sun.management package of the java.management module can be granted to all code on the class path via the option

-XaddExports:java.management/sun.management=ALL-UNNAMED

The -XaddExports option must be used with great care. You can use it to gain access to an internal API of a library module, or even of the JDK itself, but you do so at your own risk: If that internal API changes or is removed then your library or application will fail.

Patching module content

When testing and debugging it is sometimes useful to replace selected class files or resources of specific modules with alternate or experimental versions, or to provide entirely new class files, resources, and even packages. This can be done via the -Xpatch option, at both compile time and run time. Its syntax is:

-Xpatch:<module>=<file>(<pathsep><file>)*

where <module> is a module name, <file> is the filesystem path name of a module definition, and <pathsep> is the host platform's path-separator character.

The -Xpatch option can be used more than once, but at most once for any particular module name. The effect of each instance is to change how the module system searches for a type in the specified module. Before it checks the actual module, whether part of the system or defined on a module path, it first checks, in order, each module definition specified to the option. A patch path names a sequence of module definitions but it is not a module path, since it has "leaky," class-path-like semantics. This allows a test harness, e.g., to inject multiple tests into the same package without having to copy all of the tests into a single directory.

The -Xpatch option cannot be used to replace module-info.class files. If a module-info.class file is found in a module definition on a patch path then a warning will be issued and the file will be ignored.

If a package found in a module definition on a patch path is not already exported by that module then it will, still, not be exported. It can be exported explicitly via either the reflection API or the -XaddExports option.

The -Xpatch option replaces the -Xbootclasspath:/p option, which has been removed (see below).

The -Xpatch option is intended only for testing and debugging.
Its use in production settings is strongly discouraged.

Compile time

The javac compiler implements the options described above, as applicable to compile time: -modulesourcepath, -upgrademodulepath, -system, -modulepath, -addmods, -limitmods, -XaddReads, -XaddExports, and -Xpatch.

The compiler operates in one of three modes, each of which implements additional options.

In legacy mode the compiler behaves in essentially the same way as it does in JDK 8.

Single-module mode is used to compile code organized in a traditional package-hierarchical directory tree. It is the natural replacement for simple uses of legacy mode of the form

$ javac -d classes -classpath classes -sourcepath src Foo.java

If a module descriptor in the form of a module-info.java or module-info.class file is specified on the command line, or is found on the source path or the class path, then source files will be compiled as members of the module named by that descriptor and that module will be the sole root module. Otherwise if the -Xmodule:<module> option is present then source files will be compiled as members of <module>, which will be the root module. Otherwise source files will be compiled as members of the unnamed module, and the root modules will be computed as described above.

It is possible to put arbitrary classes and JAR files on the class path in this mode, but that is not recommended since it amounts to treating those classes and JAR files as part of the module being compiled.

Multi-module mode is used to compile one or more modules, whose source code is laid out in exploded-module directories on the module source path. In this mode the module membership of a type is determined by the position of its source file in the module source path, so each source file specified on the command line must exist within an element of that path. The set of root modules is the set of modules for which at least one source file is specified.

In contrast to the other modes, in this mode an output directory must be specified via the -d option. The output directory will be structured as an element of a module path, i.e., it will contain exploded-module directories which themselves contain class and resource files. If the compiler finds a module on the module source path but cannot find the source file for some type in that module then it will search the output directory for the corresponding class file.

In large systems the source code for a particular module may be spread across several different directories. In the JDK itself, e.g., the source files for a module may be found in any one of the directories src/<module>/share/classes, src/<module>/<os>/classes, or build/gensrc/<module>, where <os> is the name of the target operating system. To express this in a module source path while preserving module identities we allow each element of such a path to use braces ({ and }) to enclose commas-separated lists of alternatives and a single asterisk (*) to stand for the module name. The module source path for the JDK can then be written as

{src/*/{share,<os>}/classes,build/gensrc/*}

Packaging: Modular JAR files

The jar tool can be used without change to create modular JAR files, since a modular JAR file is just a JAR file with a module-info.class file in its root directory.

The jar tool implements the following new options to allow the insertion of additional information into module descriptors as modules are packaged:

Additionally, the new --print-module-descriptor option, or -p for short, will display the module descriptor, if any, of an existing JAR file.

The jar tool's --help option can be used to get a complete summary of its command-line options.

Packaging: JMOD files

The new JMOD format goes beyond JAR files to include native code, configuration files, and other kinds of data that do not fit naturally, if at all, into JAR files. JMOD files are used to package the modules of the JDK itself; they can also be used by developers to package their own modules, if desired. The final format of JMOD files is an open issue, but for now it is based on ZIP files.

JMOD files can be used at compile time and link time, but not at run time. To support them at run time would require, in general, that we be prepared to extract and link native-code libraries on-the-fly. This is feasible on most platforms, though it can be very tricky, and we have not seen many use cases that require this capability, so for simplicity we have chosen to limit the utility of JMOD files in this release.

A new command-line tool, jmod, can be used to create JMOD files and list the content of existing files. Its general syntax is:

$ jmod (create|list|describe) <options> <jmod-file>

The list and describe subcommands accept no options; for the create subcommand, <options> can include the --main-class, --module-version, --hash-modules, and --modulepath options described above for the jar tool, and also:

The jmod tool's --help option can be used to get a complete summary of its command-line options.

The details of the command-line linking tool, jlink, are described in JEP 282. At a high level its general syntax is:

$ jlink <options> --modulepath <modulepath> --output <path>

where the --modulepath option specifies the set of observable modules to be considered by the linker and the --output option specifies the path of the directory that will contain the resulting run-time image. The other <options> can include the --limitmods and --addmods options, described above, as well as additional linker-specific options.

The jlink tool's --help option can be used to get a complete summary of its command-line options.

Run time

The HotSpot virtual machine implements the options described above, as applicable to run time: -upgrademodulepath, -modulepath, -addmods, -limitmods, -XaddReads, -XaddExports, and -Xpatch. These options can be passed to the command-line launcher, java, and also to the JNI invocation API.

The additional options specific to this phase and supported by the launcher are:

Additional diagnostic options supported by the launcher include:

The stack traces generated for exceptions at run time have been extended to include, when present, the names and version strings of relevant modules. The detail strings of exceptions such as ClassCastException, IllegalAccessException, and IllegalAccessError have also been updated to include module information. Work on similar enhancements to other types of diagnostic information is underway.

An extended example

Suppose we have an application module, com.foo.bar, which depends upon a library module, com.foo.baz. If we have the source code for both modules in the module-path directory src:

src/com.foo.bar/module-info.java
src/com.foo.bar/com/foo/bar/Main.java
src/com.foo.baz/module-info.java
src/com.foo.baz/com/foo/baz/BazGenerator.java

then we can compile them, together:

$ javac -modulesourcepath src -d mods $(find src -name '*.java')

The output directory, mods, is a module-path directory containing exploded, compiled definitions of the two modules:

mods/com.foo.bar/module-info.class
mods/com.foo.bar/com/foo/bar/Main.class
mods/com.foo.baz/module-info.class
mods/com.foo.baz/com/foo/baz/BazGenerator.class

Assuming that the com.foo.bar.Main class contains the application's entry point, we can run these modules as-is:

$ java -mp mods -m com.foo.bar/com.foo.bar.Main

Alternatively, we can package them up into modular JAR files:

$ jar --create -f mlib/com.foo.bar-1.0.jar \
      --main-class com.foo.bar.Main --module-version 1.0 \
      -C mods/com.foo.bar .
$ jar --create -f mlib/com.foo.baz-1.0.jar \
      --module-version 1.0 -C mods/com.foo.baz .

The mlib directory is a module-path directory containing the packaged, compiled definitions of the two modules:

$ ls -l mlib
-rw-r--r-- 1501 Sep  6 12:23 com.foo.bar-1.0.jar
-rw-r--r-- 1376 Sep  6 12:23 com.foo.baz-1.0.jar

We can now run the packaged modules directly:

$ java -mp mlib -m com.foo.bar

jtreg enhancements

The jtreg test harness supports a new declarative tag, @modules. It takes a series of arguments, each of which can be of the form <module> or <module>/<package>. In either case the test will only be run when the named module is present. The latter case indicates, additionally, that the test requires access to the named package of that module; if the package is not exported then the harness will arrange for it to be exported to the module that contains the test.

A default set of @modules arguments, which will be used for all tests in a directory hierarchy that do not include such a tag, can be specified as the value of the modules property in a TEST.ROOT file or in any TEST.properties file.

The existing @compile tag accepts a new option, /module=<module>. This has the effect of using the -Xmodule option to javac, defined above, to compile the specified classes as members of the named module.

Class loaders

The Java SE Platform API historically specified two class loaders: The bootstrap class loader, which loads classes from the bootstrap class path, and the system class loader, which is the default delegation parent for new class loaders and, typically, the class loader used to load and start the application. The specification does not mandate the concrete types of either of these class loaders, nor their precise delegation relationship.

The JDK has, since the 1.2 release, implemented a three-level hierarchy of class loaders, where each loader delegates to the next:

JDK 9 retains this three-level hierarchy, in order to preserve compatibility, while making the following changes to implement the module system:

The platform class loader is retained not only for compatibility but, also, to improve security. Types loaded by the bootstrap class loader are implicitly granted all security permissions (AllPermission), but many of these types do not actually require all permissions. We have de-privileged modules that do not require all permissions by defining them to the platform class loader rather than the bootstrap class loader, and by granting them the permissions they actually need in the default security policy file. The Java SE and JDK modules defined to the platform class loader are:

java.activation
java.annotations.common
java.compact1
java.compact2
java.compact3
java.compiler
java.corba
java.scripting
java.se
java.se.ee
java.sql
java.sql.rowset
java.transaction
java.xml.bind
java.xml.ws
jdk.accessibility
jdk.charsets
jdk.crypto.ec
jdk.crypto.pkcs11
jdk.dynalink
jdk.jsobject
jdk.localedata
jdk.naming.dns
jdk.scripting.nashorn
jdk.xml.dom
jdk.zipfs

All other Java SE and JDK modules are defined to the bootstrap class loader except for the JDK modules that provide tools or export tool APIs, which are defined to the application class loader. These are:

jdk.attach
jdk.compiler
jdk.hotspot.agent
jdk.internal.le
jdk.internal.opt
jdk.jartool
jdk.javadoc
jdk.jconsole
jdk.jdeps
jdk.jdi
jdk.jlink
jdk.jshell
jdk.jstatd
jdk.jvmstat

The three built-in class loaders work together to load classes as follows:

The application and platform class loaders delegate to their respective parent loaders in order to ensure that the bootstrap class path is still searched when a class is not found in a module defined to one of the built-in loaders.

Removed: Bootstrap class-path options

In earlier releases the -Xbootclasspath option allows the default bootstrap class path to be overridden, and the -Xbootclasspath/p option allows a sequence of files and directories to be prepended to the default path. The computed value of this path is reported via the JDK-specific system property sun.boot.class.path.

With the module system in place the bootstrap class path is empty by default, since bootstrap classes are loaded from their respective modules. The javac compiler only supports the -Xbootclasspath option in legacy mode, the java launcher no longer supports either of these options, and the system property sun.boot.class.path has been removed.

The compiler's -system option can be used to specify an alternate source of system modules, as described above, and its -release option can be used to specify an alternate platform version, as described in JEP 247 (Compile for Older Platform Versions). At run time the -Xpatch option, mentioned above, can be used to inject content into modules in the initial module graph.

A related option, -Xbootclasspath/a, allows files and directories to be appended to the default bootstrap class path. This option, and the related API in the java.lang.instrument package, is sometimes used by instrumentation agents, so for compatibility it is still supported at run time. Its value, if specified, is reported via the JDK-specific system property jdk.boot.class.path.append. This option can be passed to the command-line launcher, java, and also to the JNI invocation API.

Open design issues

Testing

Many existing tests will be affected by the introduction of the module system. In JDK 9 the @modules tag, described above, has already been added to over 3,000 unit and regression tests, and many tests that used the -Xbootclasspath/p option or assumed that the system class loader is a URLClassLoader have been updated.

There is, of course, an extensive set of unit tests for the module system itself. In the JDK 9 source forest most of the run-time tests are in the test/jdk/jigsaw directory of the jdk repository and the runtime/modules directory of the hotspot repository; most of the compile-time tests are in the tools/javac/modules directory of the langtools repository.

Early-access builds containing the changes described here have been available for some time. We encourage members of the wider Java community to test their tools, libraries, and applications against these builds to help tease out any remaining compatibility issues.

Risks and Assumptions

The primary risks of this proposal are ones of compatibility due to changes to existing language constructs, APIs, and tools.

Changes due primarily to the introduction of the Java Platform Module System (JSR 376) include:

Modules that define Java EE APIs, or APIs primarily of interest to Java EE applications, are not resolved by default for code on the class path:

The run-time behavior of some Java SE APIs has changed, though in ways that continue to honor their existing specifications:

There is one source-incompatible Java SE API change:

Finally, changes due to revisions to JDK-specific APIs and tools include:

As with the introduction of modular images, it is impossible to determine the full impact of these changes in the abstract. We must therefore rely upon extensive internal and—especially—external testing. If some of these changes prove to be insurmountable hurdles for developers, deployers, or end users then we will investigate ways to mitigate their impact.

Dependences

JEP 200 (The Modular JDK) originally defined the modules present in the JDK in an XML document, as an interim measure. This JEP moved those definitions to proper module descriptors, i.e., module-info.java and module-info.class files, and the modules.xml file in the root source-code repository was removed.

The initial implementation of JEP 220 (Modular Run-Time Images) in JDK 9 used a custom build-time tool to construct JRE and JDK images. This JEP replaced that tool with the jlink tool.

Modular JAR files can also be Multi-Release JAR files, per JEP 238.