|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Chapter 4
Running JCK 6b TestsJCK 6b tests work in a variety of computing environments and can be difficult to set up and run in your environment. To reduce the difficulty of setting up a test run, the JavaTest harness uses the Configuration Editor with the JCK 6b configuration interview provided by the test suite to produce the configuration information required to run the tests. Running JCK 6b tests consists of the following basic operations: Refer to 4.4 Configuration Interview for a description of the basic structure of the configuration interviews provided by the JCK test suites. In some cases, tests require special setup procedures before they can be run. Refer to 4.5 Special Setup Instructions for a list of these tests. In addition to special setup procedures, some tests require multiple runs using different configurations. Refer to 4.6 Special Test Execution Requirements for a list of these tests. 4.1 Starting the JavaTest HarnessYou can use the JavaTest harness GUI or the JavaTest harness command-line interface to run tests, write test reports, and audit tests. While most tasks can be performed in either the JavaTest harness GUI or the command-line interface, some special tasks, such as creating configurations, can only be performed by using the JavaTest harness GUI. Note - While running the tests under win32, java.io API test failures can occur if a working directory is kept on the remote drive. These failures occur when file caching functionality is included with the remote file system access services. To avoid the dependence on a network file access services, whenever possible keep your working directory on a local system where you run the JCK tests. If you see java.io test failures reporting errors (such as the data read is not that previously written or a file length is not the expected length), move the work directory to the local drive or use different software (or software configuration) to access the remote drives. Run the JavaTest harness on a computer with a minimum of 256 megabytes of physical memory. Use an agent with the JavaTest harness to run test programs on platforms with limited amounts of memory. If you are running a large test suite (10,000 or more tests), you must allocate a large amount of memory to the Java virtual machine (VM) to run the JavaTest harness properly. When running a large test suite, allocate 512 megabytes of memory to the VM machine by including -Xmx512m in the command string used to start the JavaTest harness. Note - When allocating memory to the VM, do not exceed the actual amount of memory available on your system. In addition to allocating memory to the VM heap, additional memory is required for the Java HotSpot virtual machine if you run tests in single-VM mode. Memory in the Java HotSpot virtual machine is organized into three generations: a young generation, an old generation, and a permanent generation. The permanent generation holds objects that the VM finds convenient to have the garbage collector manage, such as objects describing classes and methods, as well as the classes and methods themselves. The default value for the maximum size of the permanent generation memory (-XX:MaxPermSize=n) is platform-dependent. For 32-bit platforms, use 128mb when running tests in single-VM mode. This value is set with the following initial and maximum settings: -XX:PermSize=128m -XX:MaxPermSize=128m For 64-bit platforms, use 256 m for permanent-generation memory when running tests in single-VM mode. This value is set with the following initial and maximum, settings: -XX:PermSize=256m -XX:MaxPermSize=256m 4.1.1 Starting the JavaTest Harness GUITo start the GUI, perform any one of the following actions:
When starting the GUI from a command line, you can use additional commands to specify work directory and configuration values. See the JavaTest Harness User's Guide: Command-Line Interface document included with this release for detailed information about using additional commands and options to specify configuration values in the command line. 4.1.2 Using the JavaTest Harness Command-Line InterfaceIf you use a configuration template or a configuration generated in the JavaTest harness GUI, you can perform many tasks (such as modifying configuration files, specifying configuration values, running tests, writing reports, and auditing tests) from the JavaTest harness command-line interface that are performed in the GUI. Refer to the JavaTest Harness User's Guide: Command-Line Interface document included with this release or to the JavaTest harness online help. 4.1.3 Using the JavaTest Harness Quick Start WizardThe JavaTest harness includes a Quick Start wizard (wizard) that enables users to quickly create combinations of test suite, work directory, and configuration files and open them in properly configured Test Manager windows. The harness opens the wizard when one of the following conditions occur:
When you start the JavaTest harness for the first time, or choose to start the JavaTest harness with a new desktop, the harness displays the wizard. The wizard allows you to either specify the test suite, work directory, and configuration loaded in the GUI or to continue work on a previous test run. If you use a configuration template or load an incomplete configuration, the wizard opens the configuration editor for you to complete the configuration before using the JavaTest harness. 4.2 Generating a ConfigurationThe JavaTest harness uses the Configuration Editor and the test suite configuration interview to generate a configuration that contains the values required to run tests on your test system. See 4.4 Configuration Interview for a description of the configuration interview. Each JCK test suite provides a configuration interview that only contains the configuration questions required to collect information for running the tests of that test suite. By filtering unnecessary configuration information, each interview simplifies the process of creating a valid configuration for your test environment. After the configuration is complete, the JavaTest harness generates a Question Log that lists each question and answer in the configuration interview. You can open the Question Log from the Test Manager. See 4.4.5 Configuration Question Log for additional information. The harness saves the completed configuration and uses it to perform all test runs associated with the work directory. Note - To temporarily change configuration values, you can set the configuration values in the command line or in the command file used to start the JavaTest harness. Refer to the JavaTest Harness User's Guide: Command-Line Interface or the JavaTest harness online help. See Appendix D for a basic description of using the JavaTest harness. Detailed information about the JavaTest harness can be found in the JavaTest Harness User's Guide: Graphical User Interface and the JavaTest Harness User's Guide: Command-Line Interface included with this release or in the JavaTest harness online help. The JavaTest harness online help provides an extensive search capability that you can use to locate specific words or phrases in the online documentation. 4.2.1 Configuration TemplatesYou can create configuration templates from complete or partially completed configurations. When creating new configurations, you can use a configuration template to substantially reduce the number of values that must be provided in the interview. You can also use the Configuration Editor bookmarks feature to hide the questions in a configuration template that do not require changes to their values. Using the bookmarks feature can substantially simplify the process of completing a configuration interview. Refer to the JavaTest Harness User's Guide: Graphical User Interface or the JavaTest harness online help for information about creating and using configuration templates and bookmarks. JCK is shipped with a set of Templates that allow you to significantly shorten the time for configuring a test platform for certain JavaTest modes by filling in all necessary parameters. Note - Parameters that depend on your configuration are left blank and need to be set manually. All shipped templates belong to Sun's reference implementations and, therefore, contain parameters specific to this JDK realization. Each of the Runtime, Compiler, and Devtools test suite contains 6 templates:
The templates are located in the following subfolders in the JCK Test Suite installation folder:
4.2.1.1 Using a TemplateThe following procedures describe how to load template files.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Loading a template
in the Quick Start wizard|
Question |
Answer |
|---|---|
|
Advanced Mode — Do you wish to use the Advanced mode? |
Yes — To run the interview in Advanced mode No — To run the interview in Simple mode |
4.4.1.1 Simple Mode Default Values
In Simple mode a number of interview questions are hidden because they use answers initialized with default values.
Table 4-2 lists the default values that are set in Simple mode for the JCK Runtime test suite. The questions are listed in the order in which they appear in Advanced mode.
Table 4-2 Interview Answers — Default Values in Simple Mode for JCK Runtime Test Suite
4.4.1.2 Specify Tests to Run
You can specify the tests that are run either in the command line or in the Configuration Editor. If you specify a test folder, all tests located hierarchically beneath it are automatically selected for you. See the JavaTest Harness User's Guide: Graphical User Interface, the JavaTest Harness User's Guide: Command-Line Interface, or the JavaTest harness online help for a detailed description of how to specify the tests that are run.
In the interview, select the tests to run by providing the answers from Table 4-3.
Table 4-3 Interview Answers That Configure The Tests to Run
|
Question |
Answer |
|---|---|
|
Specify Tests to Run — Do you wish to run only selected sections of the test suite? |
Yes — To specify the tests to run No — To run all tests |
4.4.2 Test Environment Questions
Note - The configuration interview displays test environment related questions only if you run the tests and the interview in Advanced mode. See 4.4.1 Configuration Modes for more information on Advanced mode.
The JCK 6b test suites can be run in many different test environments. The first section of each interview contains a series of sub-interviews that collect the information required to configure a test environment.
To successfully pass the tests in the JCK 6b test suites, you must set the following options in the Test Execution section of each interview:
-
Use the Verifier Option question to enable full class file verification in the configuration used to run the tests.
To enable full class file verification, set the -Xfuture option in the Verifier Option field of the JCK interview.
-
When you select Yes in the Test Platform MultiJVM question, use the Other Options question to specify the extension directory that extends the tested platform for testing the pluggability of locales provided by an implementation.
Note - To test the pluggability of locales provided by an implementation, a tested platform must be extended by the JAR files in jck/lib/extensions each time it is started.
For Sun implementations, set the following option in the Other Options field:
-Djava.ext.dirs=jck/lib/extensions: java-home/lib/ext:jre-home/lib/ext
If you do not select Yes in the Test Platform MultiJVM question you cannot use the Other Options question to specify the extension directory. When you start an agent (from a command line or a browser) to run tests, specify jck/lib/extensions as an additional extension directory.
4.4.2.1 JCK Runtime Test Suite
The interview presents a series of questions and uses their answers to create a configuration for the test environment of the JCK 6b runtime test suite. The following sections describe the test environments that might be configured for the test run:
Note - The illustrations and descriptions in the following sections describe the interactions of the various VMs and systems involved in running the tests. In addition, some tests require the use of other servers, possibly on additional systems. For simplicity these servers and systems are not included here. These remote systems are described in 4.4.2.1 JCK Runtime Test Suite and in Chapter 5.
Multiple VMs Without an Agent
In the interview, when you provide the answers from Table 4-4 or from Table 4-5, the Configuration Editor creates a configuration that uses multiple VMs without an agent to run the runtime tests.
Table 4-4 Interview Answers (Runtime Running Locally) That Configure Multiple VMs Without an Agent
|
Question |
Answer |
|---|---|
|
Test Execution Mode — Select the test execution mode |
MultiJVM |
|
Testing Locally — Are you running these tests on the same system used for this interview? |
Yes |
Table 4-5 Interview Answers (Runtime Running Remotely) That Configure Multiple VMs Without an Agent
|
Question |
Answer |
|---|---|
|
Test Execution Mode — Select the test execution mode |
MultiJVM |
|
Testing Locally — Are you running these tests on the same system used for this interview? |
No |
|
Test Platform — Which operating system is used on the test platform? |
Any choice but Other |
In the test environment illustrated in Figure 4-1, the JavaTest harness and the JCK tests run in separate Java technology runtimes. The harness starts a new VM for every test. This test environment does not use an agent.
Figure 4-1 Runtime Test Environment - Multiple VMs Without an Agent
Note - If the harness displays errors indicating that classes are not found, check that the classpaths were set correctly when configuring the classpath for each of the VMs.
Multiple VMs With an Agent
In the interview, when you provide the answers to questions listed in Table 4-6, the Configuration Editor creates a configuration that uses multiple VMs with an agent for running the runtime tests.
Table 4-6 Interview Answers That Configure Multiple VMs With an Agent
|
Question |
Answer |
|---|---|
|
Test Execution Mode — Select the test execution mode |
MultiJVM |
|
Testing Locally — Are you running these tests on the same system used for this interview? |
No |
|
Test Platform — Which operating system is used on the test platform? |
Other |
In the test environment illustrated in Figure 4-2, the JavaTest harness runs on a separate computer (using one of the reference VMs) while an agent runs the tests in separate VMs on the system under test. The agent, JCK tests, and the runtime product run in separate Java runtime environments on the target system. The harness starts a new VM for every test. See 4.4.2.4 Using Agents for a description of the types of agents that you can use.
Figure 4-2 Runtime Test Environment — Multiple VMs With an Agent
Note - If the harness displays errors indicating that classes are not found, check that the class paths were set correctly when configuring the classpath for each of the VMs.
Multiple VMs Group Without an Agent
Table 4-7 Interview Answers That Configure Multiple VMs Group Without an Agent
|
Question |
Answer |
|
Test Execution Mode — Select the test execution mode |
MultiJVM group |
|
Testing Locally — Are you running these tests on the same system used for this interview? |
Yes |
MutliJVM group mode is intended to reduce the overhead of starting VMs by sharing VM instances among groups of test executions. In this mode the harness starts an internal agent VM on the system that JavaTest is run on and passes a group of the test execution to this agent. Once the internal agent completes the test group run, it closes. The new internal agent VM is started for the next group of tests. The JavaTest harness starts socket server in particular port to communicate with internal agents running on the same system. By default, the port number is chosen from the list of available ports but you can override this parameter.
Figure 4-3 Runtime Test Environment — Multiple VMs Group Without an Agent
Note - If you have a firewall installed on your system, make sure it allows incoming connections from the local host.
The internal agent waits for a test execution request from the JavaTest harness. If no test request received during a timeout, the agent VM is automatically closed.
The configuration of the MutliJVM group test execution is the same as the configuration of the MutliJVM mode except for several parameters listed in Table 4-8 that control the MutliJVM group mode execution.
Note - These parameters are not visible in the JCK configuration interview and can only be changed by setting corresponding test environments variables via the JavaTest harness command line interface. For example:
javatest -jar lib/javatest.jar -Ejck.env.runtime.testExecute.groupMode.serverPort=8080
This variable sets TCP port used for listening for incoming connections from internal agents.
See Additional Setup Commands chapter in the JavaTest Harness Command-Line Interface User's Guide.
Table 4-8 Environment variables that control execution of the MutliJVM group mode
|
Variable |
Description |
|
jck.env.runtime.testExecute.groupMode.groupSize |
Controls a number of tests executed as a group in the same agent VM instance started by the JavaTest harness. Setting groupSize value to >1 turns the group mode on. Setting groupSize to 1 turns the group mode off. Setting groupSize value to 0 removes limitation on the number of tests executed by one VM (Default is 100). Default group size provides optimal balance between performance and stability of JCK runs. |
|
jck.env.runtime.testExecute.groupMode.serverPort |
A TCP port used for listening for incoming connections from internal agents. Default value is 0, which means that JVM will automatically choose a free port on the test system. You can specify a particular port number to avoid firewall issues. |
|
jck.env.runtime.testExecute.groupMode.timeout |
Timeout (in seconds) between test executions during which an agent is active. Default is 60 seconds. Increase this value if necessary. |
|
jck.env.runtime.testExecute.groupMode.agentArgs |
Additional agent command line arguments passed to internal agents while starting. For example, you can specify "-trace" option to enable diagnostic output from the agent. |
|
jck.env.runtime.testExecute.groupMode.closeIfFail |
A policy used to close agents after test execution if a test status is fail. Default is false, i.e. unset. |
|
jck.env.runtime.testExecute.groupMode.keepOnError |
A policy used to close agents after test execution if a test status is error. Default is false, i.e. unset. |
|
jck.env.runtime.testExecute.groupMode.debug |
Setting this variable enables additional logging. By default debug is off. |
Note - When you select the MutliJVM group mode, JCK does its best to optimize execution by grouping tests and executing them in the same VM instance. However, in case tests cannot be grouped (for example, tests marked by the following keywords: "interactive", "jdwp", "jplis", "jvmti", "jplislivephase", "jvmtilivephase", "jniinvocationapi", "only_once"), the tests will be executed individually.
Multiple VMs Group With an Agent
Table 4-9 Interview answers that configure Multiple VMs group with an agent
|
Question |
Answer |
|
Test Execution Mode — Select the test execution mode |
MultiJVM group |
|
Testing Locally — Are you running these tests on the same system used for this interview? |
No |
|
Test Platform |
Other |
When you answer interview questions as specified in Table 4-9, the JavaTest harness runs on a separate computer in the test environment (using one of the reference VMs) while an agent runs tests in separate VMs system under test. The agent, JCK tests, and the runtime product run in separate Java runtime environments on the target system. The agent starts a new internal agent VM on the target system and passes a group of test execution to this agent.
Figure 4-4 Runtime Test Environment — Multiple VMs Group With an Agent
All MultiJVM group mode configuration parameters, mentioned in Table 4-8 apply to this execution mode.
See 4.4.2.4 Using Agents for a description of the types of agents that you can use.
Single VM With an Agent
In the interview, when you select SingleJVM in the following question, the Configuration Editor creates a configuration using a single VM with an agent to run the runtime tests:
Test Execution Mode — Select the test execution mode
In the test environment illustrated in Figure 4-5, the JavaTest harness runs on a separate computer (using one of the Sun reference VMs), while an agent and the JCK tests run on your runtime product located on the target system.
This test environment is used when the system under test does not support multiple processes. Because the agent and the JCK tests run in the same VM, the classpath for the agent VM must be correct for both the agent and the tests. See 4.4.2.4 Using Agents for a description of the types of agents that you can use.
Note - If the harness displays errors indicating that classes are not found, check that the class paths were set correctly when configuring the class path for each of the VMs.
Figure 4-5 Runtime Test Environment — Single VM With an Agent
If your product runs in a browser on Win32 or Solaris platform, Figure 4-6 can also represent your test configuration. The dashed box in Figure 4-6 represents the browser. In the test environment illustrated in Figure 4-6, the JavaTest harness, agent and the tests run in a browser on the same system (the runtime product).
Figure 4-6 Runtime Test Environment — Single VM Browser Environment
A significant difference exists between the single and multiple VM environment configurations. In multiple VM environment configuration, all of the information required to configure and execute the Java runtime environment under test is specified in the JCK configuration file. In single VM environments, configuration information is spread across the following multiple locations:
-
In the current JCK configuration
-
On the command line for the Java runtime environment under test
-
In the HTML file used to load the agent applet (if you are testing a browser)
Because the Java runtime environment under test is only launched once for a test run, the bulk of the JCK configuration happens when the VM is started.
-
The Java technology security manager must be correctly configured to run tests.
-
The native code library path must be configured to allow the Java Native Interface (JNITM) implementation and Extra-Attribute tests to run.
-
Any necessary VM options must be specified.
-
Any other required shell environment variables must be specified before the runtime under test is started with the agent.
-
If you are not testing a browser and not using the -loadDir option, the class path must also be correctly specified to include the agent and the test classes before starting the runtime under test.
-
If you are testing a browser implementation of the Java runtime environment, you must supply the correct classpath in an HTML file.
4.4.2.2 JCK Compiler Test Suite
The interview presents a series of questions and uses their answers to create a configuration for the test environment of the JCK 6b compiler test suite. The following sections describe the possible configurations of test environments used to run the tests of the JCK compiler test suite:
It is important to note how the JCK compiler tests work. RMI compilers and the Java programming language compiler (Java compiler) compile test files and the resulting class files are executed on a Sun reference runtime implementation. To compile and execute compiler tests, you must configure five test commands that invoke the following compilers and runtime:
-
Java complier
-
RMI compiler
-
RMI Compiler v1.1
-
RMI Compiler v1.2
-
Sun reference runtime
Table 4-10 describes the different types of compiler tests.
Table 4-10 Types of JCK Compiler Tests
|
Type of Test |
Processes Started |
|---|---|
|
Negative |
Only the Java compiler is started, the compiler is expected to fail and produce an error. A new compiler is started for every test. |
|
Positive (Java compiler) |
A new Java compiler is started for every test. Output of the compiler is always checked using a new reference VM. |
|
Positive (RMI compiler) |
New Java and RMI compilers are started for every test. Output of the compilers is always checked using a new reference VM. |
Multiple VMs Without an Agent
In the interview, when you provide the answers from Table 4-11, the Configuration Editor creates a configuration that uses multiple VMs without an agent to run the compiler tests.
Table 4-11 Interview Answers (Compiler Running Locally) That Configure Multiple VMs Without an Agent
|
Question |
Answer |
|---|---|
|
Test Execution Mode — Select the test execution mode |
MultiJVM |
|
Testing Locally — Are you running these tests on the same system used for this interview? |
Yes |
In the test environment illustrated in Figure 4-7, the JavaTest harness, the compilers under test, and the reference VMs that test the compiler output are run on the same computer but in separate processes. This test environment does not use an agent. Processes are started as required depending on the type of compiler test.
Figure 4-7 Compiler Test Environment — Multiple VMs Without an Agent
Note - If the harness displays errors indicating that classes are not found, check that the classpaths were set correctly when configuring the classpath for each of the VMs.
Multiple VMs With an Agent
In the interview, when you provide the answers from Table 4-12, the Configuration Editor creates a configuration that uses multiple VMs with an agent to run the compiler tests.
Table 4-12 Interview Answers (Compiler Running Remotely) That Configure Multiple VMs With an Agent
|
Question |
Answer |
|---|---|
|
Test Execution Mode — Select the test execution mode |
MultiJVM |
|
Testing Locally — Are you running these tests on the same system used for this interview? |
No |
|
Test Platform — Which operating system is used on the test platform? |
Any choice |
In the test environment illustrated in Figure 4-8, the JavaTest harness and the reference VMs that test the compiler output run on the same computer. The agent, the Java compiler and the RMI compiler under test run on another system (the system under test) in separate processes. Processes are started as required depending on the type of compiler test. See Table 4-10 for a description of the different types of compiler tests. See 4.4.2.4 Using Agents for a description of the types of agents that you can use.
Figure 4-8 Compiler Test Environment — Multiple VMs With an Agent
Note - If the harness displays errors indicating that classes are not found, check that the classpaths were set correctly when configuring the classpath for each of the VMs.
Multiple VMs Group Without an Agent
Table 4-13 Interview answers that configure Multiple VMs group without an agent
|
Question |
Answer |
|
Test Execution Mode — Select the test execution mode |
MultiJVM group |
|
Testing Locally — Are you running these tests on the same system used for this interview? |
Yes |
MultiJVM group mode is intended to reduce the overhead of starting VMs by sharing Java complier and Sun reference runtime VM instances among groups of test executions. In this mode the harness starts an internal agent VM on the system that JavaTest is run on and passes a group of the test execution to this agent. Once the internal agent completes its test group run, it closes. The new internal agent VM is started for the next group of tests. The JavaTest harness starts socket server in particular port to communicate with internal agents running on the same system. By default, the port number is chosen from the list of available ports but you can override this parameter.
Figure 4-9 Compiler Test Environment — Multiple VMs Group Without an Agent
Note - If you have a firewall installed on your system, make sure it allows incoming connections from the local host.
The internal agent waits for a test execution request from the JavaTest harness. If no test request received during a timeout, the agent VM is automatically closed.
Note - The MultiJVM group is not applied to RMI compiler. It is applied only to Java compiler and Sun reference runtime.
The MultiJVM group mode expects Java compiler under test to be configured to run via the Java Compiler API interface. The configuration of the MultiJVM group mode is the same as the configuration of the MultiJVM mode except for several parameters listed in Table 4-14 that control the MultiJVM group mode execution.
Note - These parameters are not visible in the JCK configuration interview and can only be changed by setting corresponding test environments variables via the JavaTest harness command line interface. For example:
javatest -jar lib/javatest.jar -Ejck.env.compiler.testCompile.groupMode.serverPort=8080
This variable sets TCP port used for listening for incoming connections from internal agents.
See Additional Setup Commands chapter in the JavaTest Harness Command-Line Interface User's Guide.
Table 4-14 Environment variables that control execution of the MutliJVM group mode
|
Variable |
Description |
|
jck.env.compiler.testCompile.groupMode.groupSize |
Controls a number of tests executed as a group in the same agent VM instance started by the JavaTest harness. Setting groupSize value to >1 turns the group mode on. Setting groupSize to 1 turns the group mode off. Setting groupSize value to 0 removes limitation on the number of tests executed by one VM (Default is 100). Default group size provides optimal balance between performance and stability of JCK runs. |
|
jck.env.compiler.testCompile.groupMode.serverPort |
A TCP port used for listening for incoming connections from internal agents. Default value is 0, which means that JVM will automatically choose a free port on the test system. You can specify a particular port number to avoid firewall issues. |
|
jck.env.compiler.testCompile.groupMode.timeout |
Timeout (in seconds) between test executions during which the agent is active. Default is 60 seconds. Increase this value if necessary. |
|
jck.env.compiler.testCompile.groupMode.agentArgs |
Additional agent command line arguments passed to internal agents while starting. For example, you can specify "-trace" option to enable diagnostic output from the agent. |
|
jck.env.compiler.testCompile.groupMode.closeIfFail |
A policy used to close agents after test execution if a test status is fail. Default is false, i.e. unset. |
|
jck.env.compiler.testCompile.groupMode.keepOnError |
A policy used to close agents after test execution if a test status is error. Default is false, i.e. unset. |
|
jck.env.compiler.testCompile.groupMode.debug |
Setting this variable enables additional logging. By default debug is off. |
Table 4-15 Environment variables that control execution of the MutliJVM group mode for the Sun's reference runtime
|
Variable |
Description |
|
jck.env.compiler.compRefExecute.groupMode.groupSize |
Controls a number of tests executed as a group in the same agent VM instance started by the JavaTest harness. Setting groupSize value to >1 turns the group mode on. Setting groupSize to 1 turns the group mode off. Setting groupSize value to 0 removes limitation on the number of tests executed by one VM (Default is 100). Default group size provides optimal balance between performance and stability of JCK runs. |
|
jck.env.compiler.compRefExecute.groupMode.serverPort |
A TCP port used for listening for incoming connections from internal agents. Default value is 0, which means that JVM will automatically choose a free port on the test system. You can specify a particular port number to avoid firewall issues. |
|
jck.env.compiler.compRefExecute.groupMode.timeout |
Timeout (in seconds) between test executions during which the agent is active. Default is 60 seconds. Increase this value if necessary. |
|
jck.env.compiler.compRefExecute.groupMode.agentArgs |
Additional agent command line arguments passed to internal agents while starting. For example, you can specify "-trace" option to enable diagnostic output from the agent. |
|
jck.env.compiler.compRefExecute.groupMode.closeIfFail |
A policy used to close agents after test execution if a test status is fail. Default is false, i.e. unset. |
|
jck.env.compiler.compRefExecute.groupMode.keepOnError |
A policy used to close agents after test execution if a test status is error. Default is false, i.e. unset. |
|
jck.env.compiler.compRefExecute.groupMode.debug |
Setting this variable enables additional logging. By default debug is off. |
Note - When you select the MutliJVM group mode, JCK does its best to optimize execution by grouping tests and executing them in the same VM instance. However, in case tests cannot be grouped (for example, tests marked by the keyword "only_once"), the tests will be executed individually.
Multiple VMs Group With an Agent
Table 4-16 Interview answers that configure Multiple VMs group with an agent
|
Question |
Answer |
|
Test Execution Mode — Select the test execution mode |
MultiJVM group |
|
Testing Locally — Are you running these tests on the same system used for this interview? |
No |
|
Test Platform |
Other |
When you answer interview questions as specified in Table 4-16, the JavaTest harness runs on a separate computer in the test environment (using one of the reference VMs) while an agent runs tests in separate VMs system under test. The agent, JCK tests, and the compiler run in separate Java runtime environments on the target system. The agent starts a new internal agent VM on the target system and passes a group of test execution to this agent. However, the reference runtime is started on the system that JavaTest is run.
Figure 4-10 Compiler Test Environment — Multiple VMs Group With an Agent
All Multi JVM group mode configuration parameters, mentioned in Table 4-14 apply to this execution mode. See 4.4.2.4 Using Agents for a description of the types of agents that you can use.
Single VM With an Agent
In the interview, when you select SingleJVM in the following question, the Configuration Editor creates a configuration that uses a single VM with an agent to run the compiler tests:
Test Execution Mode — Select the test execution mode
In the test environment illustrated in Figure 4-11, the JavaTest harness and the reference VM that tests the compiler output are run on the same computer. An agent, the Java compiler, and the RMI compiler run in the same process on another system (the system under test). This test configuration is used when the test system does not support multiple processes.
The configuration for single VM test commands that use an agent to invoke a compiler is different from the multiple VM commands. To achieve a better performance, the command, executing reference runtime, is the same as in the Multi JVM group mode in which the harness starts an internal agent VM on the system that JavaTest is run on and passes a group of the test execution to this agent. You can change this to use Multi JVM mode by setting the environment variable jck.env.compiler.compRefExecute.groupMode.groupSize to 1. See Table 4-15 Environment variables that control execution of the MutliJVM group mode for the Sun's reference runtime for more information.
Because the agent and the compilers run in the same VM the classpath for the agent VM must be correct for the agent and the compilers. See 4.4.2.4 Using Agents for a description of the types of agents that you can use.
Figure 4-11 Compiler Test Environment — Single VM With an Agent
Note - If the harness displays errors indicating that classes are not found, check that the classpaths were set correctly when configuring the class path for each of the VMs.
4.4.2.3 JCK Devtools Test Suite
The interview presents a series of questions and uses their answers to create a configuration for the test environment of the JCK devtools (devtools) test suite. The following sections describe the possible configurations of test environments used to run the tests of the devtools test suite:
It is important to note the following descriptions of how the devtools tests work:
-
The schema compiler compiles test files and executes the resulting class files on a Sun reference runtime implementation.
-
The schema generator generates schemas from test files and validates the resulting schemas on the Sun reference runtime implementation.
Both the schema compiler and the schema generator are types of compilers. The schema compiler maps from schemas to Java class files. The schema generator maps from Java class files to schemas.
Note - Devtools compiler is used in this section to refer to both the schema compiler and the schema generator.
To compile and execute devtools compiler tests, you must configure three test commands that invoke the following:
-
Schema compiler
-
Schema generator
-
Sun reference runtime
Table 4-17 describes the different types of devtools compiler tests.
Table 4-17 Types of Devtools Compiler Tests
|
Type of Test |
Processes Started |
|---|---|
|
Negative (Schema Compiler) |
Only the Schema compiler is started, the compiler is expected to fail and produce an error. A new compiler is started for every test. |
|
Positive (Schema Compiler) |
A new Schema compiler is started for every test. Output of the compiler is always checked using a new reference VM. |
|
Positive (Schema Generator) |
A new Schema generator is started for every test. Output of the generator is always checked using a new reference VM. |
Multiple VMs Without an Agent
In the interview, when you provide the answers from Table 4-11, the Configuration Editor creates a configuration that uses multiple VMs without an agent to run the devtools compiler tests.
Table 4-18 Interview Answers (Devtools Compiler Running Locally) That Configure Multiple VMs Without an Agent
|
Question |
Answer |
|---|---|
|
Test Execution Mode — Select the test execution mode |
MultiJVM |
|
Testing Locally — Are you running these tests on the same system used for this interview? |
Yes |
In the test environment illustrated in Figure 4-12, the JavaTest harness, the devtools compiler under test, and the reference VMs that test the devtools compiler output are run on the same computer but in separate processes. This test environment does not use an agent. Processes are started as required depending on the type of devtools compiler test.
Figure 4-12 Devtools Compiler Test Environment — Multiple VMs Without an Agent
Note - If the harness displays errors indicating that classes are not found, check that the classpaths were set correctly when configuring the classpath for each of the VMs.
Multiple VMs With an Agent
In the interview, when you provide the answers from Table 4-12, the Configuration Editor creates a configuration that uses multiple VMs with an agent to run the devtools compiler tests.
Table 4-19 Interview Answers (Devtools Compiler Running Remotely) That Configure Multiple VMs With an Agent
|
Question |
Answer |
|---|---|
|
Test Execution Mode — Select the test execution mode |
MultiJVM |
|
Testing Locally — Are you running these tests on the same system used for this interview? |
No |
In the test environment illustrated in Figure 4-13, the JavaTest harness and the reference VMs that test the devtools compiler output run on the same computer. The agent and the devtools compilers run on another system (the system under test) and in separate processes. Processes are started as required depending on the type of compiler test. See Table 4-17 for a description of the types of compiler tests. See 4.4.2.4 Using Agents for a description of the types of agents that you can use.
Figure 4-13 Devtools Compiler Test Environment — Multiple VMs With an Agent
Note - If the harness displays errors indicating that classes are not found, check that the classpaths were set correctly when configuring the classpath for each of the VMs.
Multiple VMs Group Without an Agent
Table 4-20 Interview answers (Devtools Compiler Running Locally) that configure Multiple VMs group without an agent
|
Question |
Answer |
|
Test Execution Mode — Select the test execution mode |
MultiJVM group |
|
Testing Locally — Are you running these tests on the same system used for this interview? |
Yes |
MultiJVM group mode is intended to reduce the overhead of starting VMs by sharing Sun reference complier and Sun reference runtime VM instances among groups of devtools test executions. In this mode the harness starts an internal agent VM on the system that JavaTest is run on and passes a group of the test execution to this agent. Once the internal agent completes its test group run, it closes. The new internal agent VM is started for the next group of tests. The JavaTest harness starts socket server in particular port to communicate with internal agents running on the same system. By default, the port number is chosen from the list of available ports but you can override this parameter.
Figure 4-14 Devtools Compiler Test Environment — Multiple VMs Group Without an Agent
Note - If you have a firewall installed on your system, make sure it allows incoming connections from the local host.
The internal agent waits for a test execution request from the JavaTest harness. If no test request received during a timeout, the agent VM is automatically closed.
Note - The MultiJVM group is not applied to Devtools compilers. It is applied only to Sun reference runtime and Sun reference Java compiler.
Note - In this mode, the interview question about the path to Sun reference java compiler will not be asked. The reference compiler will be automatically configured to be run via Java Compiler API interface using Sun reference runtime.
The configuration of the MultiJVM group mode is the same as the configuration of the MultiJVM mode except for several parameters listed in Table 4-14 that control the MultiJVM group mode execution.
Note - These parameters are not visible in the JCK configuration interview and can only be changed by setting corresponding test environments variables via the JavaTest harness command line interface. For example:
javatest -jar lib/javatest.jar -Ejck.env.devtools.testExecute.groupMode.groupSize=400 This variable sets the number of tests grouped in one VM instance.
See Additional Setup Commands chapter in the JavaTest Harness Command-Line Interface User's Guide.
Table 4-21 Environment variables that control execution of the MultiJVM group mode
|
Variable |
Description |
|
jck.env.devtools.testExecute.groupMode.groupSize |
Controls a number of tests executed as a group in the same agent VM instance started by the JavaTest harness. Setting groupSize value to >1 turns the group mode on. Setting groupSize to 1 turns the group mode off. Setting groupSize value to 0 removes limitation on the number of tests executed by one VM (Default is 100). Default group size provides optimal balance between performance and stability of JCK runs. |
|
jck.env.devtools.testExecute.groupMode.serverPort |
A TCP port used for listening for incoming connections from internal agents. Default value is 0, which means that JVM will automatically choose a free port on the test system. You can specify a particular port number to avoid firewall issues. |
|
jck.env.devtools.testExecute.groupMode.timeout |
Timeout (in seconds) between test executions during which an agent is active. Default is 60 seconds. Increase this value if necessary. |
|
jck.env.devtools.testExecute.groupMode.agentArgs |
Additional agent command line arguments passed to internal agents while starting. For example, you can specify "-trace" option to enable diagnostic output from the agent. |
|
jck.env.devtools.testExecute.groupMode.closeIfFail |
A policy used to close agents after test execution if a test status is fail. Default is false, i.e. unset. |
|
jck.env.devtools.testExecute.groupMode.keepOnError |
A policy used to close agents after test execution if a test status is error. Default is false, i.e. unset. |
|
jck.env.devtools.testExecute.groupMode.debug |
Setting this variable enables additional logging. By default debug is off. |
Note - When you select the MutliJVM group mode, JCK does its best to optimize execution by grouping tests and executing them in the same VM instance. However, in case tests cannot be grouped (for example, tests marked by the keyword "only_once"), the tests will be executed individually.
Multiple VMs Group With an Agent
Table 4-22 Interview answers (Devtools Compiler Running Remotely) that configure Multiple VMs group with an agent
|
Question |
Answer |
|
Test Execution Mode — Select the test execution mode |
MultiJVM group |
|
Testing Locally — Are you running these tests on the same system used for this interview? |
No |
When you answer interview questions as specified in Table 4-22, the JavaTest harness runs on a separate computer in the test environment (using one of the reference VMs) while an agent runs tests in separate VMs system under test. The agent, JCK tests, and the devtools compiler run in separate Java runtime environments on the target system. The agent starts a new internal agent VM on the target system and passes a group of test execution to this agent. However, the reference runtime is started on the system that JavaTest is run.
Figure 4-15 Devtools Compiler Test Environment — Multiple VMs Group With an Agent
All Multi JVM group mode configuration parameters, mentioned in Table 4-21 apply to this execution mode. See 4.4.2.4 Using Agents for a description of the types of agents that you can use.
Single VM With an Agent
In the interview, when you select SingleJVM in the following question, the Configuration Editor creates a configuration that uses a single VM with an agent to run the devtools compiler tests:
Test Execution Mode — Select the test execution mode
Note - In this test environment, the harness uses an agent when running tests locally or remotely.
In the test environment illustrated in Figure 4-16, the JavaTest harness and the reference VM that tests the compiler output are run on the same computer while the agent and the compilers under test run on another system (the system under test) in the same VM. This test configuration is used when the test system does not support multiple processes.
The configuration for single VM test commands that use an agent to invoke a compiler is different from the multiple VM commands. To achieve better performance, the command, executing reference runtime and reference compiler, is the same as in the Multi JVM group mode, in which the harness starts an internal agent VM on the system that JavaTest is run on and passes a group of the test execution to this agent. You can change this to use Multi JVM mode by setting the environment variable jck.env.devtools.testExecute.groupMode to 1. See chapter Multiple VMs Group Without an Agent for more information about the group mode.
Because the agent and the compilers run in the same VM, the classpath for the agent VM must be correct for both the agent and the compilers. See 4.4.2.4 Using Agents for a description of the types of agents that you can use.
Figure 4-16 Devtools Compiler Test Environment — Single VM With an Agent
Note - If the harness displays errors indicating that classes are not found, check that the classpaths were set correctly when configuring the classpath for each of the VMs.
4.4.2.4 Using Agents
When you configure your test environment, you specify whether you use an active or passive agent to run the tests. Active agents initiate the connection with the JavaTest harness, as opposed to passive agents that require the JavaTest harness to initiate the connection.
If you are using the JavaTest harness agent provided by the JavaTest harness, see the JavaTest Agent User's Guide or the JavaTest harness online help for detailed instructions about configuring and using JavaTest harness agents to run tests.
If you are using another agent, see the documentation provided with that agent for detailed instructions about configuring and using the agent to run tests.
4.4.3 Test-Specific Questions
The second section of each interview contains sub interviews that collect configuration information about the tests to be run. Each interview uses the More Info panel of the configuration editor to provide specific test configuration information based on your answers to previous interview questions.
For additional information about special setup and configuration requirements for running specific tests in the test suite, see Chapter 5.
4.4.4 Test Run Questions
The last section of each interview specifies how to run the tests. For example, you can run tests concurrently, set the time factor, and include or exclude tests from the test run by using keywords and using exclude lists.
See the JavaTest Harness User's Guide: Graphical User Interface included with this release or the JavaTest harness online help for detailed instructions about using the configuration editor window.
Tip - In Simple mode, theTest Run Questions section of the configuration is hidden. You can access these values via the JavaTest GUI using the Quick Set Mode interview.
4.4.4.1 Exclude Lists
In the command line or in the GUI configuration editor window, you can specify the exclude lists used when running tests. You can choose the exclude list provided by the JCK 6b test suite or you can run the tests without using an exclude list. If your system has Internet access, you can also configure the JavaTest harness to automatically check for and download updated JCK 6b exclude lists before running tests. See the JavaTest Harness User's Guide: Graphical User Interface, the JavaTest Harness User's Guide: Command-Line Interface, or the JavaTest harness online help for a detailed description of specifying one or more exclude lists when running tests.
4.4.4.2 Keywords
Each test can contain one or more keywords defined by the JCK test suite that can be used to restrict the set of tests that are run. Keywords can also be associated with different groups of tests within the test suite. In the command line or in the GUI configuration editor window, you can specify the keywords used to filter the tests that are run. See the JavaTest Harness User's Guide: Graphical User Interface, the JavaTest Harness User's Guide: Command-Line Interface, or the JavaTest harness online help for a detailed description specifying keywords used when running tests. See A.1.3 Keywords for more information about keywords.
4.4.5 Configuration Question Log
When the Configuration Editor saves a configuration, it generates a Configuration Question Log that lists each question asked, the tag name of the question, and the corresponding answer.
The Configuration Question Log can be used during troubleshooting to quickly review the contents of the current configuration interview.
See the JavaTest Harness User's Guide: Graphical User Interface or the JavaTest harness online help for a detailed description of using the Configuration Question Log.
4.4.6 Configuration Checklist
When the Configuration Editor saves a configuration, it generates a Configuration Checklist from each interview. Some tests require additional setup steps (such as starting additional programs) prior to running them. You can use the Configuration Checklist to verify that all required preparations are complete before running tests.
Note - Because the Configuration Editor generates the checklist from the completed interview, it only lists those setup steps actually required by that configuration to run tests.
See the JavaTest Harness User's Guide: Graphical User Interface or the JavaTest harness online help for a detailed description of using the Configuration Checklist.
4.5 Special Setup Instructions
Special setup instructions are described on a test-by-test basis in Chapter 5 as well as in the Configuration Checklist. See 4.4.6 Configuration Checklist for information about the Configuration Checklist. Table 4-23 lists the tests that require additional setup and the page containing the setup instructions.
Table 4-23 Special Test Setup Instructions
|
Test |
Section and Page Number |
|---|---|
4.6 Special Test Execution Requirements
The following tests require multiple runs using different configurations:
VM Tests — Test specific information required to run the VM tests in all required configurations is provided in the VM tests Execution section of Chapter 5.
4.7 JCK-devtools Test Suite Execution Requirements
This section includes special considerations, necessary steps, and preliminary tasks required to run JCK-devtools test suite tests.
-
Agent classpath should contain the path to the jtjck.jar file to run Schema Compiler, Schema Generator, and JAX-WS tests.
-
When you run schema compilation through the JavaTest Agent, your schema compiler will be executed in the Agent's JVM. Your implementation should provide a custom class, which implements the following interface:
package com.sun.jck.lib;
public interface SchemaCompilerTool {
/**
* @param xsdFiles array of strings containing schema files
* @param packageName the target package
* @param outDir output directory where java file(s) will be generated
* @param out output stream for logging
* @param err error stream for logging
* @return 0 if java file(s) generated successfully
*/
int compile(String[] xsdFiles, String packageName, File outDir, PrintStream out, PrintStream err);
}
Schema compilation should be accomplished during invocation of compile method. The JCK devtools test suite includes the following class that is fully compatible with Sun's reference implementation and that can be used for schema compilation: com.sun.jck.lib.SchemaCompiler. The realization of the method compile(String[] xsdFiles, String packageName, File outDir, PrintStream out, PrintStream err) compiles one or more schemas into java sources.
-
When you run schema generation through JavaTest Agent, your generator will be executed in the Agent's JVM. Your implementation should provide a custom class, which implements the following interface:
package com.sun.jck.lib;
public interface SchemaGenTool {
int generate(String[] args, PrintStream out, PrintStream err);
}
The invocation of the generate method should start schema generation.
The JCK devtools test suite includes the following class that is fully compatible with Sun's reference implementation and that can be used for schema generation: com.sun.jck.lib.SchemaGen. The realization of the method generate(String[] args, java.io.PrintStream out, java.io.PrintStream err) generates xml schema from one or more java files.
-
JCK provides commands to invoke your schema compiler. For example, for the Sun JAXB implementation on Solaris, Sun provides a script solaris/bin/xjc.sh that compiles schemas into class files, the script can be found at the following location:
solaris/bin/xjc.sh
For the Sun JAXB implementation on Windows, Sun provides a script win32\bin\xjc.bat, which compiles schemas into class files, the command can be found at the following location:
win32\bin\xjc.bat
The xjc script supports the following mandatory command-line options:
Usage: xjc –cp <arg> –p <pkg> –d <dir> <schema file>
Options:
-
–cp <arg> — specifies class search path
-
–p <pkg> — specifies the target package
-
–d <dir> — target directory to store generated files
-
-
JCK provides commands to invoke your schema generator. For example, for the Sun JAXB implementation on Solaris, Sun provides a script solaris/bin/schemagen.sh that generates schemas from java files, the script can be found at the following location:
solaris/bin/schemagen.sh
For the Sun JAXB implementation on Windows, Sun provides a script win32\bin\schemagen.bat that compiles schemas into class files, the script can be found at the following location:
win32\bin\schemagen.bat
The schemagen script supports the following mandatory command-line options:
Usage: schemagen –cp <arg> –d <dir> <java files>
Options:
-
–cp <arg> — specifies class search path
-
–d <dir> — target directory to store generated files
-
 Previous |
Contents | Index | Next |
