JEP 281: HotSpot C++ Unit-Test Framework
|Component||hotspot / test|
|Discussion||hostspot dash dev at openjdk dot java dot net|
|Reviewed by||Aleksandre Iline, Mikael Vidstedt|
|Endorsed by||Mikael Vidstedt|
Enable and encourage the development of C++ unit tests for HotSpot.
Support writing and executing unit tests for methods, classes and subsystems
Support unit tests, where only a unit is tested and nothing else is run
Support tests which require VM initialization
Support fast tests, where the execution time is in the order of milliseconds
Support both positive and negative testing
Allow test isolation
Support tests co-located with the product source code
Support integration with the current infrastructure
Produce an individual result for each test
Be able to easily run an individual tests (from the command line)
Be able to provide minimal reproducers for test failures
Provide IDE support
Allow the framework to evolve, including making fast fixes to the framework
Support test selection and test grouping, with granularity similar to
Allow testing of any compile target, product as well as debug
Allow tests of platform-dependent code
Provide minimal documentation: A how-to wiki and examples in the repository
Allow the exclusion of tests from execution by modifying the source of the tests or other files, such as exclude lists
Support for all internal tests to be converted
Support the JDK 9 build platforms supported by Oracle
- Replace Java tests. Unit testing in C++ is a complement to testing for different use cases.
In a well-tested codebase it is easier to make changes. The test suite supports the engineer who makes the changes by verifying that nothing unexpected breaks.
Today HotSpot has many tests, but not many tests of the most direct type, and it is not easy enough to write such tests.
Introducing a test framework for C++ is the first step towards a better test suite. A test framework for C++ supports test writing in the same language as the JVM and then the internal structures are directly exposed to the test code, which gives another level of possibilities to easily write small sharp tests, compared to doing functional testing from java using
A possibility to develop unit tests for existing functionality will make it possible to test C++ code for new features in isolation and make it easier to write regression tests for some of the more esoteric issues.
The Google Test framework (GTest) is the C++ unit test framework which most closely meets our goals, and it's an xUnit test framework, with a lot of traction in the community. The GTest framework:
- Is developed and supported by others
- Provides IDE integration with Eclipse IDE
- Is a battle-proven, complete API
- Has a feature-rich execution model
- Has existing documentation and examples
- Supports JUnit-style test results, and integration with Hudson and Jenkins
Several tasks are needed to allow using GTest for writing tests for HotSpot, and some additional tasks are needed to enhance it. In the current state of GTest:
- C++ constructs are used which HotSpot does not use and which are disabled in HotSpot, e.g., exceptions, templates, and STL
- Solaris/Oracle Solaris Studio isn't a supported OS/compiler
GTest is, admittedly, a third party tool, thus adding another dependency to the existing build and test process. GTest is also fairly big (71K LOC) and may change incompatibly in the future. To avoid the risk of changes in the test framework itself leading to problems, we need to control which version of GTest is used, and be able to specify that as part of the build (though it should be possible to override). It would be beneficial to have a dependency system to automate the download and install of the correct version of GTest.
HotSpot test directory layout
The new tests need a place to live in the source tree. The root directory for the tests should be placed close to, but not in, the product source itself, much like the existing test-directory structure. For clarity the tests should not be mixed with existing
jtreg tests; instead, they should be split in two directories. We propose to split the current
jdk9/hotspot/test directory into two subdirectories:
jtreg tests will move down into the
java directory (including JNI code and and shell scripts). The
TEST.ROOT file will remain at the top level.
Build targets and binaries
The product binaries must not be affected by the test code in any visible way. For example, there should not be additional symbols exported and the product bundles should not include any tests. The compiled tests will be put in separate test bundles, one per configuration. The tests will link to symbols exported from a non-stripped JVM library, which is created from the same object files as regular library.
It must be easy to run the tests from the command line using
make. In order for the test results to be compatible with results from other tests the invocation may potentially run using a
jtreg test wrapper, which in turn invokes GTest. GTest can itself produce JUnit-style results, which integrates nicely with Hudson/Jenkins and similar tools.
Alternative 1: HUTT. A prototype framework called "HotSpot Unit Test Tool" (HUTT) was created earlier, and is an xUnit framework. It is a significantly smaller framework than the GTest is (2K LOC) and is not an external dependency. It is a viable but more expensive solution. It also lacks IDE support.
Alternative 2: Keep implementing tests in Java. It is possible to access JVM internals using a Whitebox API. Adding Whitebox APIs is cumbersome in comparison and slow in execution. It is suitable for some introspection, but far from all testing. Java tests are more costly to write and execute, since in order to get high quality tests which target specific functionality the tests become very complex and it is often hard to guarantee determinism.
Alternative 3: Keep using internal tests. This solution would not meet many of the stated goals.
Risks and Assumptions
Risk: GTest could evolve in a direction which makes it unsuitable as a HotSpot unit test framework. The risk is estimated to be low.
Mitigation plan: Fork the GTest framework, or use HUTT.
Risk: Backporting GTest fixes will prove to be very costly.
Mitigation plan: Fork the GTest framework, or use HUTT.