JDK team uses a broad range of test methodologies to develop test scenarios to qualify the JDK.
|Unit, regression testing||
Engineering writes unit tests to cover sufficient fundamental scenarios to ensure no regressions are introduced in future.
A unit test is a small test that verifies the new code is doing what its author thinks it should do. Regression tests are closely related to unit tests.
A regression test is a test written in order to verify that a change in existing code actually fixes a reported bug. Today's unit test is tomorrow's regression test.
Thousands of unit/regression tests
Java Compatibility Kit is used to assure conformance of Java SE implementation to the specification
Thousands of tests, executed in 20+ different combination
Functional tests are used to verify the functionality or behavior of the feature under test. (positive and negative tests)
Thousands of tests. Mostly automated for server side components. For client components includes manual testing for all different look and feels supported by the operating systems.
Based on available code coverage tools, Analysis at frequent interval done to identify areas to focus for additional coverage. (other factors also contribute in final decision on which areas need additional test coverage. e.g. Code churn for the area, customer importance of the area, escalation trend analysis etc.)
|Reliability, performance, stress testing||
JVM reliability and performance
is tested by tracking, executing and publicly posting of results
from running a broad range of public benchmarks:
Internal stress test tools, random data generator tools are used to stress test the JVM.
Reliability testing is also done on networking side.
|Fault Injection testing||
Verification that system responds gracefully to fault conditions and remains operational during unexpected software failures
Other than ensuring Java platform compatibility using TCKs, other client and server side compatibility testing is done.
Unscripted testing left to the tester to attempt to break a feature implementation to find bugs.
Bug festivals are arranged at important milestones where all teams are invited to come and in a given a specified duration to focus and try to break the product with with provided tests/tools.
|Customer beta programs||Apart from internal testing, targeted and public beta programs are conducted at beta milestones to gather feedback and strengthen the areas needed.|