Running tests using jtreg

There are various ways to run the unit and regression tests, varying in convenience and flexibility.

Use targets in test/Makefile

This method works well for running some standard predefined sets of tests, in a predefined manner. To run individual tests, or custom sets of tests, it is best to use the convenience scripts to run jtreg, or to run jtreg directly.

The top level repository, and the jdk and langtools repositories, in an OpenJDK forest provide Makefile targets to run standard blocks of tests. The targets all have names of the form jdk_* and langtools_*. You can see a summary of the available targets in the top level test/Makefile.

When using these targets, it is recommended to set the following variables, either as environment variables or on the "make" command line:

  • JT_HOME: the jtreg installation directory
  • PRODUCT_HOME: the JDK build to be tested: this is typically a file path ending in j2sdk-image
Use convenience scripts to run jtreg

This method provides the most flexibility for specifying which tests are to be run, and how to run them. It is particularly well suited to running individual tests to help debug an issue in the platform being tested.

There is very little magic in the script, and anything described in this section can be equivalently achieved by running jtreg.jar directly.

In the following list, recommended options are shown with a pale green background. Other options which may be worth consideration are shown with a pale yellow background.

  • To specify which version of jtreg to use, add the appropriate jtreg-dir/platform/bin directory to your execution path.

    Note: the "platform" component is preserved for historical reasons. The three supported variants ("solaris", "win32" and "linux") are now all equivalent, and suitable for use on Mac OS X as well.

  • If you are using jtreg to run tests on a new version of JDK, it can be safer to use a different, stable version of JDK to run jtreg itself. However, unless your new version of JDK is badly broken, it is usually good enough to use the new version of JDK to run jtreg as well.

    If you want to explicitly specify the JDK used to run jtreg, set the JT_JAVA environment variable before running jtreg. Otherwise, leave it unset, and jtreg will default to using the same version of JDK to run jtreg as is used to execute the tests.

  • To execute one or more tests, you must minimally specify the version of JDK to use, and which tests to be run:

    % jtreg -jdk:test-jdk test-or-folder...

    test-jdk will be used to run the tests in the specified files and/or folders. It will also be used to run jtreg unless you have set the JT_JAVA environment variable, as described above.

  • jtreg uses two directories in which to store results, creating them if necessary. The work directory is used to store the compiled classes for tests, temporary files while tests are running, and the detailed results of each test. The report directory is used to write HTML and plain text reports at the end of the test run.

    By default, these directories are named JTwork and JTreport, and are placed in the current directory used when jtreg is run.

    It is common to execute jtreg in top level directory of the repository in which you are working, such as the jdk or langtools repositories. It is also common to place the work and report directories in a subdirectory of the main build directory, using the -w and -r options. For example,

    % jtreg -w build-dir/jtreg/work -r build-dir/jtreg/report ...
  • To vary the amount of output written to the console for each test, use the -verbose:arg option.

  • To enable assertions while running tests, use the standard JVM options -ea:arg and -esa, or their equivalents.

  • Some tests in the test suite may have problems such that the tests are currently deemed dysfunctional or invalid, and have therefore been disabled. This may be done in two ways.

    A test may be directly labelled with @ignore. By default, these tests will cause an error if you try to run them. To avoid running them, use the -ignore:quiet option. (You can also ignore the @ignore directives by using the -ignore:run option, but this is not common.)

    Tests may also be listed in a problem list or exclude list file. This is a more flexible mechanism than @ignore and allows the test to be listed as problematic on specific platforms. To avoid running these tests, use the -exclude:file option.

  • By default, jtreg runs each test in its own JVM. While this provides maximum isolation between tests, it also incurs maximum overhead. jtreg provides different modes for running tests, trading off isolation for performance. Many groups of tests can be run in the faster modes, and work is ongoing to upgrade additional tests. In general, it is recommended to use -agentvm to run the tests in "agentvm" mode. See jtreg: old and new for a blog entry giving details of the different execution modes.

  • Some tests, typically GUI tests, may require manual interaction in order to determine whether they pass or fail. To run just those tests, use the -m or -manual option. To avoid running such tests, use the -a or -automatic option.

  • The tests in some parts of the test suite may be able to run concurrently. This feature is only available in the "othervm" and "agentvm" modes. To have jtreg run tests concurrently, use the -conc:value option, where value is either the maximum number of tests that should be allowed to run concurrently or the word "auto" to set the value to the number of processors on your system. Make sure you have enough memory and/or swap space when you use this option. For more information, see Running tests concurrently with jtreg.

  • If you have built a component of JDK such as the compiler (javac), you may be able to test it in conjunction with a complete JDK by putting the classes or jar file for new component on the system bootstrap classpath:

    % jtreg -jdk:jdk -Xbootclasspath/p:component.jar test-or-folder...
  • You can rerun any selection of tests. Use the -status:arg option to filter the set of tests to be run, based on their prior result status. For example, to rerun tests which previously failed or could not be run:

    % jtreg -jdk:jdk -status:fail,error test-or-folder...
  • For info on all the available options, use the -help option. For info on a specific option or topic, use -help option-or-word... To read the online help, use the -onlineHelp command line option, or start jtreg with the -gui option, and use the Help menu.

  • Here is an example that puts many of the preceding ideas together. JDK 6 is used to run jtreg, while jtreg uses JDK 8 to run the java/lang/Class tests in the jdk repository located at /w/jjg/work/tl/jdk. The tests are executed using the "agentvm" execution mode; a one line summary for each test is written as each test completes, and the overall results are stored in the local build directory.

    export JT_JAVA=/opt/jdk/1.6.0
    cd /w/jjg/work/tl/jdk
    /opt/jtreg/4.1/linux/bin/jtreg \
        -jdk:/opt/jdk/1.8.0 -agentvm -verbose:summary \
        -w build/jtreg/work -r build/jtreg/report \
Run jtreg.jar directly

Instead of using the startup scripts provided, you can also execute the jtreg.jar file directly, or you can run jtreg from Ant. See the online help for details.

Test Results

Summary reports are generated in the report directory, in plain text and HTML formats. To generate a detailed report in XML as well, use the -xml option.

Detailed results for each test are stored in the work directory, in .jtr files. These are plain text files that are easy enough to read directly if needed.

Individual tests can be in one of four states.