JEP 281: HotSpot C++ Unit-Test Framework

AuthorStefan Sarne
OwnerIgor Ignatyev
Created2014/06/24 11:35
Updated2017/04/10 04:24
TypeFeature
StatusClosed / Delivered
Componenthotspot / test
ScopeImplementation
Discussionhostspot dash dev at openjdk dot java dot net
EffortM
DurationM
Priority2
Reviewed byAleksandre Iline, Mikael Vidstedt
Endorsed byMikael Vidstedt
Release9
Issue8047975

Summary

Enable and encourage the development of C++ unit tests for HotSpot.

Goals

Non-Goals

Motivation

In a well-tested codebase it is easier to make changes. The test suite supports the engineer who makes the changes by verifying that nothing unexpected breaks.

Today HotSpot has many tests, but not many tests of the most direct type, and it is not easy enough to write such tests.

Introducing a test framework for C++ is the first step towards a better test suite. A test framework for C++ supports test writing in the same language as the JVM and then the internal structures are directly exposed to the test code, which gives another level of possibilities to easily write small sharp tests, compared to doing functional testing from java using jtreg.

A possibility to develop unit tests for existing functionality will make it possible to test C++ code for new features in isolation and make it easier to write regression tests for some of the more esoteric issues.

Description

The Google Test framework (GTest) is the C++ unit test framework which most closely meets our goals, and it's an xUnit test framework, with a lot of traction in the community. The GTest framework:

Several tasks are needed to allow using GTest for writing tests for HotSpot, and some additional tasks are needed to enhance it. In the current state of GTest:

GTest is, admittedly, a third party tool, thus adding another dependency to the existing build and test process. GTest is also fairly big (71K LOC) and may change incompatibly in the future. To avoid the risk of changes in the test framework itself leading to problems, we need to control which version of GTest is used, and be able to specify that as part of the build (though it should be possible to override). It would be beneficial to have a dependency system to automate the download and install of the correct version of GTest.

HotSpot test directory layout

The new tests need a place to live in the source tree. The root directory for the tests should be placed close to, but not in, the product source itself, much like the existing test-directory structure. For clarity the tests should not be mixed with existing jtreg tests; instead, they should be split in two directories. We propose to split the current jdk9/hotspot/test directory into two subdirectories:

Existing jtreg tests will move down into the java directory (including JNI code and and shell scripts). The TEST.ROOT file will remain at the top level.

Build targets and binaries

The product binaries must not be affected by the test code in any visible way. For example, there should not be additional symbols exported and the product bundles should not include any tests. The compiled tests will be put in separate test bundles, one per configuration. The tests will link to symbols exported from a non-stripped JVM library, which is created from the same object files as regular library.

Invoking tests

It must be easy to run the tests from the command line using make. In order for the test results to be compatible with results from other tests the invocation may potentially run using a jtreg test wrapper, which in turn invokes GTest. GTest can itself produce JUnit-style results, which integrates nicely with Hudson/Jenkins and similar tools.

Alternatives

Alternative 1: HUTT. A prototype framework called "HotSpot Unit Test Tool" (HUTT) was created earlier, and is an xUnit framework. It is a significantly smaller framework than the GTest is (2K LOC) and is not an external dependency. It is a viable but more expensive solution. It also lacks IDE support.

Alternative 2: Keep implementing tests in Java. It is possible to access JVM internals using a Whitebox API. Adding Whitebox APIs is cumbersome in comparison and slow in execution. It is suitable for some introspection, but far from all testing. Java tests are more costly to write and execute, since in order to get high quality tests which target specific functionality the tests become very complex and it is often hard to guarantee determinism.

Alternative 3: Keep using internal tests. This solution would not meet many of the stated goals.

Risks and Assumptions

Risk: GTest could evolve in a direction which makes it unsuitable as a HotSpot unit test framework. The risk is estimated to be low.

Mitigation plan: Fork the GTest framework, or use HUTT.

Risk: Backporting GTest fixes will prove to be very costly.

Mitigation plan: Fork the GTest framework, or use HUTT.