An Introduction to jtreg
What is jtreg?
jtreg is the primary test harness that is used for running unit and regression tests for JDK. It was started back in 1997, and currently supports over 10,000 tests that have been written for JDK in the years since then.
The early tests for the Java platform were written to test the HotJava browser, and consisted of a series of web pages that had to be manually visited and the instructions followed. That led to the development of conformance tests and the JavaTest test harness, which together formed the Java Compatibility Kit, or JCK. But after the initial introduction of JCK, it became clear we needed a different sort of test harness for running regression tests. JCK tests are defined in a declarative style: "this is a runtime test", "this is an RMI compiler test", "this is a negative compiler test", and the test harness knows how to run the different types of test. But for regression tests, we needed a more imperative style of test: "compile this, then compile that, then run this class file". This led to the jtreg Tag Specification. In addition, the JavaTest test harness began its evolution from "the test harness for JCK" into a framework for creating test harnesses, for test suites such as JCK, and now, as jtharness, for the JDK regression test suite
What were the requirements for jtreg?
The primary requirement was that it should be easy for any JDK developer to create and run tests. It was also a strong requirement that it should be easy to run any test outside the test harness, so that it would be easy to debug the test and/or debug any problems in the JDK that a test might uncover.
It was also a requirement that the harness should be able to support different types of tests, for different JDK components, such as the runtime, compiler, and support for applets (remember: this was back in 1997.)
Being able to test the runtime meant that the harness had to be able to support testing all aspects of the API, including System.exit, security managers, and system properties.
And finally, it was a requirement that the harness had to be flexible in accomodating new types of tests. For example, there is an important category of tests for the compiler involving separate compilation: compile a number of files, then modify some, and finally, try to recompile them.
Why not JUnit?
Back in 1997, JUnit was not around. And neither was Ant, which is one of the reasons why jtreg provides basic support for compiling test classes as well as running them. In addition, we needed a test harness capable of testing javac, support for applets, and other components of JDK, as well as simply testing the runtime API.
What sort of tests does jtreg support?
jtreg supports various different types of tests:
- API tests: In general, these have a main program and which pass if the main program returns normally, or fail if the program throws an exception. In addition, support has also been added recently for including TestNG tests within a jtreg test suite.
- Compiler tests: These consist of one or more source files which either should compile (a positive test) or should not compile (a negative test).
- Applet tests: These consist of an HTML file containing an applet tag, and source files for the corresponding applet class. Some amount of manual interaction is required to determine if the test passed or failed.
- Shell tests: These consist of a shell script and any additional files as needed. They can be used to create tests that do not easily fall into any of the preceding categories. Such a test passes if and only if the script exits with a 0 return code.
What does a jtreg test look like?
A test is one or more source files. One of the files should contain a special comment, called a "test description", which provides details about the test and how it should be executed. The test is identified by the path of the file containing the test description, relative to the top level directory of the test suite, which is identified by it containing a file called TEST.ROOT. TEST.ROOT may optionally contain some overall configuration values for the test suite.
What's in a "test description"?
A test description gives information about a test, and how to run it. It is stored in a comment block near the head of a test's main source file. It always begins with "@test".
/* * @test * @bug 1234567 * @summary description of the test * @run main MyTest */
This test description consists of the initial prefix
@test), followed by a couple of informational tags
@summary) which are used to
track the reason for the test, and an action tag (
main) which specifies the action to take to run this test,
namely, run the class
Source files for simple tests are automatically compiled when necessary; there is no need to precompile test classes with tools like Ant or make. For more complex tests, you may need specify the names of additional classes which may be required when running the test.
How does jtreg run tests?
A test description contains one or more action tags. (If none are found, a default one is assumed, appropriate to the type of file.) The actions are executed sequentially until one fails or until all have been executed. The test is considered to have passed if and only if all its constituent actions passed.
There are three different modes to execute actions that involve a JVM.
- Any action run in this mode is run in its own JVM. This provides the maximum isolation between the actions, but also the maximum overhead, so it is the slowest of these three modes.
- All actions are run in the same JVM. This is the fastest mode, but also has the maximum risk of interference between actions (tests). In the worst case, one bad test can cause all the following tests in the test run to fail.
- "Like samevm, but better". This mode is more robust than samevm: actions are run in a shared JVM, called an Agent. After each action completes, the agent will attempt to reset the JVM to a clean state; if it cannot, the agent will be shutdown and a new one created when next required. In the best case, the performance is similar to samevm mode; in the worst case it degrades to being similar to othervm mode.
In general, the default mode is selected on the command line, but it can be overriden for a directory of tests, or for an individual action of a test.
In othervm and agentvm modes, jtreg can run tests concurrently, using multiple JVMs. Care must still be taken to ensure that tests do not interfere with each other using external resources such as absolute filenames (like /tmp) or fixed network ports. Since tests can and often will manipulate global data within a JVM (such as the system properties, streams, or even the security manager), it is never appropriate to run tests concurrently within the same JVM.