JEP 230: Microbenchmark Suite

AuthorsClaes Redestad, Staffan Friberg, Aleksey Shipilev
OwnerClaes Redestad
TypeFeature
ScopeJDK
StatusTargeted
Release12
Componentperformance
Discussionjdk dash dev at openjdk dot java dot net
EffortS
DurationM
Reviewed byJoe Darcy, Mikael Vidstedt
Endorsed byMikael Vidstedt
Created2014/07/17 00:07
Updated2018/11/12 17:29
Issue8050952

Summary

Add a basic suite of microbenchmarks to the JDK source code, and make it easy for developers to run existing microbenchmarks and create new ones.

Goals

Non-Goals

Description

The microbenchmark suite will be co-located with the JDK source code in a single directory and, when built, will produce a single JAR file. Co-location will simplify adding and locating benchmarks during development. When running benchmarks, JMH provides powerful filtering capabilities that allow the user to run only the benchmarks that are currently of interest. The exact location remains to be determined.

Benchmarking generally require comparisons to an earlier build, or even release, so the microbenchmarks must support JDK(N), for benchmarks targeting features in the new JDK, and JDK(N-1), for benchmarks targeting features existing in an earlier release. This means, for JDK 12, that the structure and build scripts must support compiling benchmarks for both JDK 12 and JDK 11. The benchmarks will further be divided by using Java package names that describe the area of the JDK they are testing.

The following directory structure is proposed:

jdk/jdk
   .../make/test (Shared folder for Makefiles)
   .../test (Shared folder for functional tests)
      .../micro/classes/org/openjdk/bench
         .../java (subdirectories similar to JDK packages and modules)
         .../vm (subdirectories similar to HotSpot components)
      .../micro/resources/... (resources)

The building of the microbenchmark suite will be integrated with the normal JDK build system. It will be a separate target that is not executed during normal JDK builds in order to keep the build time low for developers and others not interested in building the microbenchmark suite. To build the microbenchmark suite the user will have to specifically run make build-microbenchmark or similar. Additionally running benchmarks using make run-test TEST="micro:regexp" will be supported.

The benchmarks will all depend on JMH in much the same way that some unit tests depend on TestNG or jtreg, so while the dependency on JMH is new there are other parts of the build that have similar dependencies. One difference compared to jtreg is that JMH is both used during the build and is packaged as part of the resulting JAR file.

A Maven POM file might be added to the microbenchmark directory to enable easy development using IDEs. The target directory when using the POM file will be created under the regular JDK build directory to avoid mixing source code with the build output.

The set of benchmarks in the microbenchmark suite will be imported from of the JMH JDK Microbenchmarks project.[2] These constitute a set of tuned and tested microbenchmarks that is already in use internally. An open question is whether to migrate this standalone project in its entirety to the co-located suite, or to keep it as a stabilization forest for more long-lived regression tests.

Any user, however, is still expected to make sure that other parameters such as the execution machine and the JDK are stable and comparable when doing analysis. Benchmarks are expected, in the common case, to be able to finish a complete run in less than a minute. This is not a wrapper framework for large or long-running benchmarks; the goal is to provide a suite of fast and targeted benchmarks. In some exceptional cases benchmarks may require longer warmup or runtime in order to achieve stable results, but that should be avoided as much as possible. It is not a goal of the suite to act as a general wrapper for larger workloads; the intent, rather, is to extract a critical component or method from a larger benchmark and stress only that part as a microbenchmark.

As part of this project a new page on wiki.openjdk.java.net will be created to help explain how to develop new benchmarks and describe what the requirements are for adding a benchmark. The requirements will mandate adherence to coding standards, reproducible performance, and clear documentation of the benchmark and what it is measuring.

Alternatives

Keep maintaining the microbenchmark suite as a separate project[2].

Co-location simplifies adding benchmarks for new features, especially in a world where a large fraction of all new feature development is done in project repositories (Valhalla, Amber etc). A case that has proven to be especially convoluted in the separate project model is to test changes to javac itself, which then requires an explicit rebuild of the benchmark suite using each respective JDK. Co-locating would solve this particular use case more elegantly, while not prohibiting use pre-built benchmark bundles for performance tracking of stable tests over longer time periods.

Testing

The microbenchmarks will be validated by the performance team as part of regular performance testing to ensure that only stable, tuned, and accurate microbenchmarks are added. Evaluation and profiling will also be done for benchmarks on a case-by-case basis to ensure that it tests the intended functionality. All tests must be run multiple times on all applicable platforms to ensure that they are stable.

Risks

Dependencies

The microbenchmark suite will depend on the Java Microbenchmark Harness version 1.12 or later.

[1] http://openjdk.java.net/projects/code-tools/jmh

[2] http://openjdk.java.net/projects/code-tools/jmh-jdk-microbenchmarks/