Code Tools: jmh
JMH is a Java harness for building, running, and analysing nano/micro/milli/macro benchmarks written in Java and other languages targetting the JVM.
The recommended way to run a JMH benchmark is to use Maven to setup a standalone project that depends on the jar files of your application. This approach is preferred to ensure that the benchmarks are correctly initialized and produce reliable results. It is possible to run benchmarks from within an existing project, and even from within an IDE, however setup is more complex and the results are less reliable.
In all cases, the key to using JMH is enabling the annotation- or bytecode-processors to generate the synthetic benchmark code. Maven archetypes are the primary mechanism used to enable this. We strongly recommend new users make use of the archetype to setup the correct environment.
Preferred Usage: Command Line
- Setting up the benchmarking project.The following
command will generate the new JMH-driven project in test
$ mvn archetype:generate \ -DinteractiveMode=false \ -DarchetypeGroupId=org.openjdk.jmh \ -DarchetypeArtifactId=jmh-java-benchmark-archetype \ -DgroupId=org.sample \ -DartifactId=test \ -Dversion=1.0
If you want to benchmark an alternative JVM language, use another archetype artifact ID from the list of existing ones, it usually amounts to replacing java to another language in the artifact ID given below. Using alternative archetypes may require additional changes in the build configuration, see the
pom.xmlin the generated project.
- Building the benchmarks. After the project is generated,
you can build it with the following Maven command:
$ cd test/ $ mvn clean install
- Running the benchmarks. After the build is done, you
will get the self-contained executable JAR, which holds your
benchmark, and all essential JMH infrastructure code:
$ java -jar target/benchmarks.jar
Run with -h to see the command line options available.
When dealing with large projects, it is customary to keep the benchmarks in a separate subproject, which then depends on the tested modules via the usual build dependencies.
While the command line approach is the suggested one, some people prefer to use IDEs. The user experience varies between different IDEs, but we will outline the general things here. Running benchmarks from the IDE is generally not recommended due to generally uncontrolled environment in which the benchmarks run.
- Setting up the benchmarking project. Some IDEs provide
the GUI to create the Maven project from the given archetype. Make
sure your IDE knows about Central archetype catalog, and look for
Alternatively, you can use the command line to generate the
benchmark project, see above.
NOTE: JMH is not intended to be used in the same way as a typical testing library such as JUnit. Simply adding the
jmh-corejar file to your build is not enough to be able to run benchmarks.
- Building the benchmarks.
Most IDEs are able to open/import Maven projects, and infer the build configuration from Maven project configuration. IDEA and Netbeans are able to build JMH benchmark projects with little to no effort. Eclipse build configuration may need to set up JMH annotation processors to run.
- Running the benchmarks. There is no direct support for
JMH benchmarks in the IDE, but one can use JMH Java API to invoke
the benchmark. It usually amounts to having the
mainmethod, which will then call into JMH. See JMH Samples for the examples of this approach. Before you run any benchmark, the project build is required. Most IDEs do this automatically, but some do require explicit build action to be added before the run: adding Maven target "
install" should help there.
Other build systems
We do not ship the build scripts for other build systems, but there are community-supported bindings to Gradle, sbt, and probably other builders, see Links section below. If you want to build with an alternative build system, you may reference the Ant sample which describes the steps to build JMH benchmark projects with Ant.
Building the "Bleeding Edge" JMH
In some cases, you want bleeding edge code which contains fixes, features, and APIs not available yet in released versions. This section explains how to get the "bleeding edge" JMH in your project.
- Check out JMH source with Mercurial:
$ hg clone http://hg.openjdk.java.net/code-tools/jmh/ jmh
- Build JMH. You can optionally skip the tests:
$ cd jmh/ $ mvn clean install -DskipTests
You only need to do this step once, and Maven will deploy JMH into Maven repo on your local machine.
If you already have the benchmark project, then it is enough to change JMH dependencies versions to
1.0-SNAPSHOT. If not, create the JMH benchmark project and change the version there.
Done! Build the benchmark project, and run it:
$ mvn clean install $ java -jar target/benchmarks.jar
Getting SupportMake sure you did this before publishing the benchmark, and/or requesting the JMH feature:
- JMH annotations Javadocs and Samples are essential reading. Follow the JMH samples to get familiar with the API, use cases, culprits, and pitfalls in building the microbenchmarks and using JMH.
- Your benchmarks should be peer-reviewed. Do not assume that a nice harness will magically free you from considering benchmarking pitfalls. We only promise to make avoiding them easier, not avoiding them completely.
- Archetypes provide the golden build configuration. Try to generate the clean JMH benchmark project and transplant the benchmark there. This is important to try when upgrading to the newer JMH versions, since the minute differences in the build configurations may attribute to the failures you are seeing.
- Current development code is usually leaner, meaner, and better. Try to run with bleeding edge JMH to see if the issue is already fixed.
- See if your question/issue was discussed already. Look around mailing list archives to see if there is already an answer.
If all these did not help, you are welcome at the JMH mailing list.