JEP 139: Enhance javac to Improve Build Speed
|Author||Magnus Ihse Bursie|
|Discussion||compiler dash dev at openjdk dot java dot net|
Reduce the time required to build the JDK and enable incremental builds by modifying the Java compiler to run on all available cores in a single persistent process, track package and class dependences between builds, automatically generate header files for native methods, and clean up class and header files that are no longer needed.
The top level goals are:
- Increase build speed by having javac use all cores and reuse javac in a server process.
- Simplify work for developers by having javac build incrementally.
This project is part of a larger effort to improve the build infrastructure of the JDK.
The improvements to javac will be internal and not available through the public api of the javac launcher. Instead an internal wrapper program called smart-javac (or sjavac for short) will house the new functionality. Eventually, when the javac wrapper features have stabilized, they can be proposed in a future JEP to be moved to the public api of javac. This would allow all Java developers to take advantage of the improvements.
This project only concerns the changes needed in javac and the new wrapper. It does not cover the changes needed in the JDK Makefiles to take advantage of these changes; those are described in JEP 138: Autoconf-Based Build System.
This project will not concern changes needed in javac to speed up Javadoc generation.
All cores should be used when compiling Java sources and there should be a speedup in build performance when using multiple cores.
The workload distributed on the different cores will not be perfectly balanced and it is known that the same work will be recomputed several times. But the changes supported by this JEP will enable us to successively improve the sharing of work between the cores within javac.
An incremental build should recompile only the changed packages and their dependencies.
After an increment build has been done, where classes or native methods were removed, the output directories should be clean; i.e., no classes or C-header files corresponding to the removed sources should remain.
Building the complete OpenJDK is unnecessarily slow. This puts an extra burden on developers and build systems. As a result, developers check out and build just a part of the source code, since the product as a whole takes too long to build.
The internal smart-javac wrapper will probably be invoked like this:
$ java -jar sjavac.jar -classpath ... -sourcepath ... -pkg '*' \ -j all -h headerdir -d outputdir
This will compile all source files found in the sourcepath with package names
matching anything (
'*') using all (
-j) cores. A database file will be
created in the (
-d) outputdir called
.javac_state which will contain all
information necessary to do a fast incremental compile, with proper cleanup of
disappearing classes and C-headers as well as correct dependency tracking.
The smart-javac wrapper implements multi core support by creating a
JavaCompiler instance for each core. The source code to be compiled is then
split into packages and randomly distributed to the JavaCompilers. If a
randomly selected package has dependencies, then these dependencies will be
compiled automatically, but not written to disk, i.e.,
an implicitly compiled dependency is later requested as part of a randomly
selected package then the implicit work is not wasted; instead the
already-compiled dependency is written to disk.
Since the initial compile does not know which packages a package depends upon,
a random distribution of work is the best we can do. Therefore
java.lang.Object will be recompiled as many times as there are cores, but
only one of the JavaCompilers will be responsible for writing
java.lang.Object to disk. Enough packages are independent of each other to
make this a workable strategy for making use of multiple cores.
When we improve javac in the future (not part of this JEP), more and more work
will be shared between the JavaCompilers and eventually we will reach the state
java.lang.Object is only compiled once.
javah will be automatically run on any class that contains native methods and
the generated C-headers will be put in the (
-h) headerdir. A new annotation
@ForceNativeHeader is used for classes that have final static primitives that
need to be exported to JNI, but no native methods.
To avoid restarting javac and losing optimizations done by the JVM, the
smart-javac wrapper supports a
-server option. This option will spawn a
background javac server so that each subsequent smart-javac wrapper invocation
that refers to the same portfile will reuse the same server.
-server arguments are:
portfile= where to store the TCP port used
logfile= where to store the output from javac, defaults to portfile+”.logfile”
stdouterrfile= where to store output from the server, defaults to portfile+”.stdouterr”
javac= the path to the javac to launch by the server, with spaces and commas replaced by
Since javac cannot currently share state between concurrent compilations, each
additional core will consume roughly as much memory as a single invocation of
javac. Improved sharing between the cores will be successively introduced after
this JEP is completed. The
-j option can be used to limit the number of cores
and thus the memory usage.
The javac server stays in memory after all compilations have finished. The server will automatically shut down and free memory and other resources after 30 seconds of inactivity.
The server runs as the same user as the normal javac compiler, and thus has the same privileges and possibility to write to the build output directory. Unlike the normal javac compiler, a compilation can be trigged by connecting to the server through a TCP port. Only the command line, and not the source code, is sent over TCP.
A potential security risk is that an attacker could add the compilation of some malicious piece of code, which would appear in the output directory. To alleviate this risk, we will use the several measures:
- Open a new TCP port each time; the port number is stored in the portfile.
- Only allow connections from localhost.
- Require a unique cookie to be presented to the server before any compilation will occur.
The cookie is a 64 bit random integer stored in the portfile. The portfile has typical temporary file permissions, i.e., only the owner is allowed to read from it or write to it.
Instead of making javac properly parallel, we could start several single-threaded compilations of different and independent java packages in parallel. This would not require any changes to javac, but it would be much harder to get the Makefiles correct, and it would not give as much speed improvement.
The build infrastructure project will test that the output of the old build system and the new build system is identical. This will make sure that the smart-javac wrapper generates identical output for the same source files.
None of the new options are going to be public, so no tests need to be added to the javac test suite, but more-specific tests for the smart-javac wrapper will be added to the langtools test directory.
Risks and Assumptions
Resulting product is incorrect
- Risk: Javac changes causes incorrect bits to be build
- Mitigation plan: Test resulting build properly
The old build makefiles will be available concurrently with the new makefiles to simplify comparing of the old and new bits.
This JEP does not depend on any other changes. It forms the basis for JEP 138: Autoconf-Based Build System, which will use this new feature in javac to speed up JDK builds. The new makefiles can use an unmodified javac, but they will not achieve the desired speedup and incremental build support unless this JEP is completed.
- Compatibility: Low impact. We will not add new arguments to javac.
- Security: Low impact. In the description section, there is a discussion about the security aspects of opening up a server for compilation jobs.
- I18n/L10n: The new smart-javac wrapper features will most likely result in a few new messages; we will not fully translate these since smart-javac is not yet part of any public api.
- Testing: Apart from the build itself, separate test cases for the different smart-javac options should be written.