JEP 230: Microbenchmark Suite
|Authors||Staffan Friberg, Aleksey Shipilev|
|Discussion||platform dash jep dash discuss at openjdk dot java dot net|
|Reviewed by||Joe Darcy, Mikael Vidstedt|
|Endorsed by||Mikael Vidstedt|
Add a basic suite of microbenchmarks to the JDK source code, and make it easy for developers to run existing microbenchmarks and create new ones.
- Based on the Java Microbenchmark Harness (JMH)
- Stable and tuned benchmarks, targeted for continuous performance
- A stable and non-moving suite after the Feature Complete milestone of a feature release, and for non-feature releases
- Support comparison to previous JDK releases for applicable tests
- Easy to add new benchmarks
- Easy to update tests as APIs and options change, are deprecated, or are removed during development
- Easy to build
- Easy to find and run a benchmark
- Support JMH updates
- Include an initial set of around a hundred benchmarks in the suite
It is not a goal to provide benchmarks for new JDK 9 features. Adding benchmarks for new features will be done as part of those projects.
It is not a goal to create a complete set of benchmarks covering everything in the JDK. The suite will continue to be expanded over time with newly-written benchmarks, or through efforts specifically targeted at extending its coverage.
It is not a goal to provide a solution for handling binary dependencies in microbenchmarks. Support for this might be added later.
The microbenchmark suite will be co-located with the JDK source code in a single repository or directory and, when built, will produce a single JAR file. A single repository will simplify adding and locating benchmarks during development. When running benchmarks, JMH provides powerful filtering capabilities that allow the user to run only the benchmarks that are currently of interest. The exact location remains to be determined.
Benchmarking generally require comparisons to an earlier build, or even release, so the microbenchmarks must support JDK(N), for benchmarks targeting features in the new JDK, and JDK(N-1), for benchmarks targeting features existing in an earlier release. This means, for JDK 9, that the structure and build scripts must support compiling benchmarks for both JDK 9 and JDK 8. The benchmarks will further be divided by using Java package names that describe the area of the JDK they are testing.
jdk9/benchmark .../make (Shared folder for Makefiles) .../micro .../src/jdk8 (same as under JDK 9) .../src/jdk9 .../jdk (subdirectories similar to JDK packages and modules) .../hotspot (subdirectories similar to HotSpot components) .../javac .../nashorn .../resources (if needed)
The building of the microbenchmark suite will be integrated with the
normal JDK build system. It will be a separate target that is not
executed during normal JDK builds in order to keep the build time low for
developers and others not interested in building the microbenchmark
suite. To build the microbenchmark suite the user will have to
make microbenchmarks or something similar. It is not
yet determined how this will best be integrated into the build system,
and it will require further discussion with those responsible for the
build system. The benchmarks will all depend on JMH in much the same way
that some unit tests depend on TestNG or jtreg, so while the dependence
on JMH is new there are other parts of the build that have similar
dependencies which can be reviewed to determine an appropriate
solution. One difference compared to jtreg is that JMH is both used
during the build and is packaged as part of the resulting JAR file.
A Maven POM file will be added to the microbenchmark directory to enable easy development using IDEs. The target directory when using the POM file will be created under the regular JDK build directory to avoid mixing source code with the build output.
All added benchmarks will be thoroughly tested and reviewed to determine that they test what they are supposed to test and return predictable results, to allow them to be used in regression-testing scenarios. All required benchmark parameters will be configured as part of the benchmarks using JMH annotations to ensure trusted and easy comparison between builds. Any user, however, is still expected to make sure that other parameters such as the execution machine and the JDK are stable and comparable when doing analysis. Benchmarks are expected, in the common case, to be able to finish a complete run in less than a minute. This is not a wrapper framework for large or long-running benchmarks; the goal is to provide a suite of fast and targeted benchmarks. In some exceptional cases benchmarks may require longer warmup or runtime in order to achieve stable results, but that should be avoided as much as possible. It is not a goal of the suite to act as a general wrapper for larger workloads such a Java EE benchmarks running on top of Glassfish; the intent, rather, is to extract a critical component or method from a larger benchmark and stress only that part as a microbenchmark.
As part of this project a new page on wiki.openjdk.java.net will be created to help explain how to develop new benchmarks and describe what the requirements are for adding a benchmark. The requirements will mandate adherence to coding standards, reproducible performance, and clear documentation of the benchmark and what it is measuring.
We considered maintaining the microbenchmark suite as a separate project and repository, but co-locating the microbenchmarks with the JDK source code and in the JDK forest is the preferred solution. Co-location simplifies adding benchmarks for new features and removing benchmarks, while still keeping the suite stable for the benchmarking of different JDK releases.
The microbenchmarks will be validated by the performance team as part of the performance testing of JDK 9 to ensure that only stable, tuned, and accurate microbenchmarks are added. Evaluation and profiling will also be done for each benchmark to ensure that it tests the intended functionality. All tests must be run multiple times on all applicable platforms to ensure that they are stable.
A JDK build will depend on a version of JMH. The JDK build system will need to be able to download or in some other way access to a binary version of JMH, and it must be possible with reasonable effort to update to a newer version of JMH when required. This should be an easy-to-solve issue since we have similar dependences for TestNG and other libraries. The JMH dependency will only be visible for someone trying to build the microbenchmark suite; other build targets do not depend on JMH.
A significant increase in build time. The benchmark suite is not required for regular developer JDK builds; it will be a separate build target.
Potential test instability in new benchmarks. Thorough review and testing of each added benchmark is required.
Increase in source-code repository size. Microbenchmarks are small, as the name suggests, but there will over time hopefully be a large set of them. The main size will be due to resources that will need to be handled outside of the repository.
Binary storage needs to be available to allow creation of certain types of benchmarks. Some benchmarks rely on external libraries or large static resources such as XML documents and source-code files used as part of the benchmark. We currently have no solution for this, but this will only limit benchmarks with this type of dependence. Small, non-binary resources might be allowed as part of the microbenchmark source code.
The microbenchmark suite will depend on the Java Microbenchmark Harness version 1.0 or later.