Running tests using jtreg
There are various ways to run the unit and regression tests, varying in convenience and flexibility.
- Use targets in test/Makefile
This method works well for running some standard predefined sets of tests, in a predefined manner. To run individual tests, or custom sets of tests, it is best to use the convenience scripts to run jtreg, or to run jtreg directly.
The top level repository, and the
langtoolsrepositories, in an OpenJDK forest provide Makefile targets to run standard blocks of tests. The targets all have names of the form
langtools_*. You can see a summary of the available targets in the top level
When using these targets, it is recommended to set the following variables, either as environment variables or on the "make" command line:
JT_HOME: the jtreg installation directory
PRODUCT_HOME: the JDK build to be tested: this is typically a file path ending in j2sdk-image
- Use convenience scripts to run jtreg
This method provides the most flexibility for specifying which tests are to be run, and how to run them. It is particularly well suited to running individual tests to help debug an issue in the platform being tested.
There is very little magic in the script, and anything described in this section can be equivalently achieved by running jtreg.jar directly.
In the following list, recommended options are shown with a pale green background. Other options which may be worth consideration are shown with a pale yellow background.
To specify which version of jtreg to use, add the appropriate jtreg-dir
/bindirectory to your execution path.
Note: the "platform" component is preserved for historical reasons. The three supported variants ("solaris", "win32" and "linux") are now all equivalent, and suitable for use on Mac OS X as well.
If you are using jtreg to run tests on a new version of JDK, it can be safer to use a different, stable version of JDK to run jtreg itself. However, unless your new version of JDK is badly broken, it is usually good enough to use the new version of JDK to run jtreg as well.
If you want to explicitly specify the JDK used to run jtreg, set the
JT_JAVAenvironment variable before running jtreg. Otherwise, leave it unset, and jtreg will default to using the same version of JDK to run jtreg as is used to execute the tests.
To execute one or more tests, you must minimally specify the version of JDK to use, and which tests to be run:
% jtreg -jdk:test-jdk test-or-folder...
test-jdk will be used to run the tests in the specified files and/or folders. It will also be used to run jtreg unless you have set the
JT_JAVAenvironment variable, as described above.
jtreg uses two directories in which to store results, creating them if necessary. The work directory is used to store the compiled classes for tests, temporary files while tests are running, and the detailed results of each test. The report directory is used to write HTML and plain text reports at the end of the test run.
By default, these directories are named
JTreport, and are placed in the current directory used when jtreg is run.
It is common to execute jtreg in top level directory of the repository in which you are working, such as the
langtoolsrepositories. It is also common to place the work and report directories in a subdirectory of the main build directory, using the
-roptions. For example,
% jtreg -w build-dir/jtreg/work -r build-dir/jtreg/report ...
To vary the amount of output written to the console for each test, use the
To enable assertions while running tests, use the standard JVM options
-esa, or their equivalents.
Some tests in the test suite may have problems such that the tests are currently deemed dysfunctional or invalid, and have therefore been disabled. This may be done in two ways.
A test may be directly labelled with
@ignore. By default, these tests will cause an error if you try to run them. To avoid running them, use the
-ignore:quietoption. (You can also ignore the
@ignoredirectives by using the
-ignore:runoption, but this is not common.)
Tests may also be listed in a problem list or exclude list file. This is a more flexible mechanism than
@ignoreand allows the test to be listed as problematic on specific platforms. To avoid running these tests, use the
By default, jtreg runs each test in its own JVM. While this provides maximum isolation between tests, it also incurs maximum overhead. jtreg provides different modes for running tests, trading off isolation for performance. Many groups of tests can be run in the faster modes, and work is ongoing to upgrade additional tests. In general, it is recommended to use
-agentvmto run the tests in "agentvm" mode. See jtreg: old and new for a blog entry giving details of the different execution modes.
Some tests, typically GUI tests, may require manual interaction in order to determine whether they pass or fail. To run just those tests, use the
-manualoption. To avoid running such tests, use the
The tests in some parts of the test suite may be able to run concurrently. This feature is only available in the "othervm" and "agentvm" modes. To have jtreg run tests concurrently, use the
valueis either the maximum number of tests that should be allowed to run concurrently or the word "
auto" to set the value to the number of processors on your system. Make sure you have enough memory and/or swap space when you use this option. For more information, see Running tests concurrently with jtreg.
If you have built a component of JDK such as the compiler (javac), you may be able to test it in conjunction with a complete JDK by putting the classes or jar file for new component on the system bootstrap classpath:
% jtreg -jdk:jdk -Xbootclasspath/p:component.jar test-or-folder...
You can rerun any selection of tests. Use the
-status:argoption to filter the set of tests to be run, based on their prior result status. For example, to rerun tests which previously failed or could not be run:
% jtreg -jdk:jdk -status:fail,error test-or-folder...
For info on all the available options, use the
-helpoption. For info on a specific option or topic, use
-help option-or-word...To read the online help, use the
-onlineHelpcommand line option, or start jtreg with the
-guioption, and use the
Here is an example that puts many of the preceding ideas together. JDK 6 is used to run jtreg, while jtreg uses JDK 8 to run the
java/lang/Classtests in the
jdkrepository located at
/w/jjg/work/tl/jdk. The tests are executed using the "agentvm" execution mode; a one line summary for each test is written as each test completes, and the overall results are stored in the local build directory.
export JT_JAVA=/opt/jdk/1.6.0 cd /w/jjg/work/tl/jdk /opt/jtreg/4.1/linux/bin/jtreg \ -jdk:/opt/jdk/1.8.0 -agentvm -verbose:summary \ -w build/jtreg/work -r build/jtreg/report \ test/java/lang/Class
- Run jtreg.jar directly
Instead of using the startup scripts provided, you can also execute the
jtreg.jarfile directly, or you can run jtreg from Ant. See the online help for details.
Summary reports are generated in the report directory, in
plain text and HTML formats. To generate a detailed report in XML
as well, use the
- The plain text report in
report-dir/text/summary.txtcontains one test result per line and is well suited for processing with tools like
- The XML report can be used with CI systems like Hudson to generate trend graphs. See Regression Test Harness and Hudson Continuous Integration System on how to do this.
Detailed results for each test are stored in the work
.jtr files. These are plain text
files that are easy enough to read directly if needed.
Individual tests can be in one of four states.
- Not run: the test has not yet been run.
- Passed: the test has been run successfully.
- Failed: the test has been run, but did not succeed.
- Error: the test could not be run for some reason.