Test Methodologies

JDK team uses a broad range of test methodologies to develop test scenarios to qualify the JDK

Unit,
regression testing
Engineering writes unit tests to cover sufficient fundamental scenarios to ensure no regressions are introduced in future.

A unit test is a small test that verifies the new code is doing what its author thinks it should do. Regression tests are closely related to unit tests.

A regression test is a test written in order to verify that a change in existing code actually fixes a reported bug. Today's unit test is tomorrow's regression test.

Thousands of unit/regression tests
Conformance testing

Java Compatibility Kit is used to assure conformance of Java SE implementation to the specification

Thousands of tests, executed in 20+ different combination
Functional testing
 Functional tests are used to verify the functionality or behavior of the feature under test. (positive and negative tests)

Thousands of tests. Mostly automated for server side components. For client components includes manual testing for all different look and feels supported by the operating systems.

Based on available code coverage tools, Analysis at frequent interval done to identify areas to focus for additional coverage. (other factors also contribute in final decision on which areas need additional test coverage. e.g. Code churn for the area, customer importance of the area, escalation trend analysis etc.)

 
Reliability, performance, stress testing

JVM reliability and performance is tested by tracking, executing and publicly posting of results from running a broad range of public benchmark
SPEC: JBB2005, Web2005, JappServer, JVM98
Volano
SwingMark
Jetstrem
SciMark

Internal stress test tools, random data generator tools are used to stress test the JVM

Reliability testing is also done on networking side
Fault Injection testing

Verification that system responds gracefully to fault conditions and remains operational during unexpected software failures
Compatibility testing
 Other than ensuring Java platform compatibility using TCKs, other client and server side compatibility testing is done.

  • consumer focused – common consumer site applets/games etc.

    • 400 applets from pogo and other sources. Run for all supported browsers for the operating system.

  • enterprise focused – commonly used sun internal and external customer applications

    • Legal agreements with other companies to use their applications in our testbed may limit availability of these tests to a company other than Sun. These are complex applications that require much setup time, runtime, and evaluation of results. Aapplications used: VolanoMark v2.5.0.9 and Tomcat Servlet Container : v5.5.9. These applications need to be run for all available VM, garbage collection options.
Ad-hoc testing

Unscripted testing left to the tester to attempt to break a feature implementation to find bugs
Bug fests

Bug festivals are arranged at important milestones where all teams are invited to come and in a given a specified duration to focus and try to break the product with with provided tests/tools.
Customer beta programs
 Apart from internal testing, targeted and public beta programs are conducted at beta milestones to gather feedback and strengthen the areas needed.