This document is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 Unported License
  Previous Contents Index Next
Chapter 4

Running JCK 6b Tests

JCK 6b tests work in a variety of computing environments and can be difficult to set up and run in your environment. To reduce the difficulty of setting up a test run, the JavaTest harness uses the Configuration Editor with the JCK 6b configuration interview provided by the test suite to produce the configuration information required to run the tests.

Running JCK 6b tests consists of the following basic operations:

Refer to 4.4 Configuration Interview for a description of the basic structure of the configuration interviews provided by the JCK test suites.

In some cases, tests require special setup procedures before they can be run. Refer to 4.5 Special Setup Instructions for a list of these tests. In addition to special setup procedures, some tests require multiple runs using different configurations. Refer to 4.6 Special Test Execution Requirements for a list of these tests.

4.1 Starting the JavaTest Harness

You can use the JavaTest harness GUI or the JavaTest harness command-line interface to run tests, write test reports, and audit tests. While most tasks can be performed in either the JavaTest harness GUI or the command-line interface, some special tasks, such as creating configurations, can only be performed by using the JavaTest harness GUI.


Note - While running the tests under win32, java.io API test failures can occur if a working directory is kept on the remote drive. These failures occur when file caching functionality is included with the remote file system access services. To avoid the dependence on a network file access services, whenever possible keep your working directory on a local system where you run the JCK tests. If you see java.io test failures reporting errors (such as the data read is not that previously written or a file length is not the expected length), move the work directory to the local drive or use different software (or software configuration) to access the remote drives.


Run the JavaTest harness on a computer with a minimum of 256 megabytes of physical memory. Use an agent with the JavaTest harness to run test programs on platforms with limited amounts of memory.

If you are running a large test suite (10,000 or more tests), you must allocate a large amount of memory to the Java virtual machine (VM) to run the JavaTest harness properly. When running a large test suite, allocate 512 megabytes of memory to the VM machine by including -Xmx512m in the command string used to start the JavaTest harness.


Note - When allocating memory to the VM, do not exceed the actual amount of memory available on your system.


In addition to allocating memory to the VM heap, additional memory is required for the Java HotSpot virtual machine if you run tests in single-VM mode. Memory in the Java HotSpot virtual machine is organized into three generations: a young generation, an old generation, and a permanent generation. The permanent generation holds objects that the VM finds convenient to have the garbage collector manage, such as objects describing classes and methods, as well as the classes and methods themselves.

The default value for the maximum size of the permanent generation memory (-XX:MaxPermSize=n) is platform-dependent. For 32-bit platforms, use 128mb when running tests in single-VM mode. This value is set with the following initial and maximum settings:

-XX:PermSize=128m -XX:MaxPermSize=128m

For 64-bit platforms, use 256 m for permanent-generation memory when running tests in single-VM mode. This value is set with the following initial and maximum, settings:

-XX:PermSize=256m -XX:MaxPermSize=256m

4.1.1 Starting the JavaTest Harness GUI

To start the GUI, perform any one of the following actions:

  • Execute the javatest.jar file.

    If you have access to a command line, you can directly execute the javatest.jar file from any directory by including its path in the command.

    On UNIX systems, use the following format:

    java -Xmx512m -XX:PermSize=256m -XX:MaxPermSize=256m -jar jt-dir/javatest.jar

    On Win32 systems, use the following format:

    java -Xmx512m -XX:PermSize=128m -XX:MaxPermSize=128m -jar jt-dir\javatest.jar


    Note - The default RI PermGen values are not large enough to perform a complete JCK run at one time. 64-bit platforms require 256m permanent generation memory, whereas 32-bit platforms require 128m permanent generation memory. Include the appropriate options in the command to run tests for your test platform.


    In the command line, jt-dir represents the path to javatest.jar. In JCK, the javatest.jar file is located in the lib directory of the test suite. If you are in the directory where the javatest.jar file is located, do not include the path in the command line.

    For example, you can use the following command to start the GUI if you are in the directory where the javatest.jar file is located and are running tests in single-VM mode on a 64-bit platform:

    java -Xmx512m -XX:PermSize=256m -XX:MaxPermSize=256m -jar javatest.jar

    When starting the GUI, you can include additional commands in the command line that specify the work directory and the configuration values used to start the GUI.

    To display a list of commands that can be used to launch the JavaTest harness GUI, type the following in a command line:

    java -jar javatest.jar -help

    See the JavaTest Harness User's Guide: Command-Line Interface document included with this release for detailed descriptions of these commands.

  • Double-click a shortcut icon.

    If you are using a GUI, your system might support double clicking the javatest.jar file shortcut icon to launch the harness.


    Note - If you choose to create and use a shortcut icon to launch the GUI, verify that your system allocates adequate memory to the VM.


  • Run a startup script.

  • Invoke the class directly.

    If you have access to a command line and the JAR file is on the classpath, you can directly invoke the class from the directory where you intend to create files and store test results.

    java com.sun.javatest.tool.Main

When starting the GUI from a command line, you can use additional commands to specify work directory and configuration values. See the JavaTest Harness User's Guide: Command-Line Interface document included with this release for detailed information about using additional commands and options to specify configuration values in the command line.

4.1.2 Using the JavaTest Harness Command-Line Interface

If you use a configuration template or a configuration generated in the JavaTest harness GUI, you can perform many tasks (such as modifying configuration files, specifying configuration values, running tests, writing reports, and auditing tests) from the JavaTest harness command-line interface that are performed in the GUI. Refer to the JavaTest Harness User's Guide: Command-Line Interface document included with this release or to the JavaTest harness online help.

4.1.3 Using the JavaTest Harness Quick Start Wizard

The JavaTest harness includes a Quick Start wizard (wizard) that enables users to quickly create combinations of test suite, work directory, and configuration files and open them in properly configured Test Manager windows. The harness opens the wizard when one of the following conditions occur:

  • The harness cannot restore an existing desktop.

  • A user includes a command-line option in the command to start the GUI.

  • A user opens a new Test Manager window.

When you start the JavaTest harness for the first time, or choose to start the JavaTest harness with a new desktop, the harness displays the wizard. The wizard allows you to either specify the test suite, work directory, and configuration loaded in the GUI or to continue work on a previous test run.

If you use a configuration template or load an incomplete configuration, the wizard opens the configuration editor for you to complete the configuration before using the JavaTest harness.

4.2 Generating a Configuration

The JavaTest harness uses the Configuration Editor and the test suite configuration interview to generate a configuration that contains the values required to run tests on your test system. See 4.4 Configuration Interview for a description of the configuration interview.

Each JCK test suite provides a configuration interview that only contains the configuration questions required to collect information for running the tests of that test suite. By filtering unnecessary configuration information, each interview simplifies the process of creating a valid configuration for your test environment.

After the configuration is complete, the JavaTest harness generates a Question Log that lists each question and answer in the configuration interview. You can open the Question Log from the Test Manager. See 4.4.5 Configuration Question Log for additional information.

The harness saves the completed configuration and uses it to perform all test runs associated with the work directory.


Note - To temporarily change configuration values, you can set the configuration values in the command line or in the command file used to start the JavaTest harness. Refer to the JavaTest Harness User's Guide: Command-Line Interface or the JavaTest harness online help.


See Appendix D for a basic description of using the JavaTest harness. Detailed information about the JavaTest harness can be found in the JavaTest Harness User's Guide: Graphical User Interface and the JavaTest Harness User's Guide: Command-Line Interface included with this release or in the JavaTest harness online help. The JavaTest harness online help provides an extensive search capability that you can use to locate specific words or phrases in the online documentation.

4.2.1 Configuration Templates

You can create configuration templates from complete or partially completed configurations. When creating new configurations, you can use a configuration template to substantially reduce the number of values that must be provided in the interview.

You can also use the Configuration Editor bookmarks feature to hide the questions in a configuration template that do not require changes to their values. Using the bookmarks feature can substantially simplify the process of completing a configuration interview. Refer to the JavaTest Harness User's Guide: Graphical User Interface or the JavaTest harness online help for information about creating and using configuration templates and bookmarks.

JCK is shipped with a set of Templates that allow you to significantly shorten the time for configuring a test platform for certain JavaTest modes by filling in all necessary parameters.


Note - Parameters that depend on your configuration are left blank and need to be set manually.


All shipped templates belong to Sun's reference implementations and, therefore, contain parameters specific to this JDK realization. Each of the Runtime, Compiler, and Devtools test suite contains 6 templates:

  • Single VM With an Agent for Solaris, Linux, and Windows platforms

  • Multiple VMs With an Agent for Solaris, Linux, and Windows platforms

The templates are located in the following subfolders in the JCK Test Suite installation folder:

  • jck/solaris/multiJVM-agent-Solaris-RI.jti

  • jck/solaris/singleJVM-agent-Solaris-RI.jti

  • jck/linux/multiJVM-agent-Linux-RI.jti

  • jck/linux/singleJVM-agent-Linux-RI.jti

  • jck/win32/multiJVM-agent-WindowsXP-RI.jti

  • jck/win32/singleJVM-agent-Windows-RI.jti

4.2.1.1 Using a Template

The following procedures describe how to load template files.

Loading a template in the Quick Start wizard
  1. In JavaTest Harness click File > Open Quick Start Wizard on the main menu.
  2. Select Start a new test run and click Next.
  3. Select Use a configuration template and specify the path to the template file or click Browse to locate the template. Click Next to continue.
Loading a template in the Configuration Editor
  1. Run Configuration Editor.
  2. In the configuration editor click File > Load on the main menu

4.2.2 Using Sample Test Suites

Sample test suites are provided with the test suite for you to use in creating and testing a configuration before you attempt to run the full JCK test suite. The sampleJCK-compiler-6b, sampleJCK-runtime-6b, and sampleJCK-devtools-6b test suites contain samples of the different types of tests in the full JCK test suite. To use the sample test suites, install one of the sample test suite files.

After you start the JavaTest harness, load the sampleJCK-compiler-6b, sampleJCK-runtime-6b, or sampleJCK-devtools-6b test suite, and then complete the appropriate configuration. After you complete the configuration, test it by running the tests in the sample test suite. When you are able to successfully run the tests, you can use this configuration to run the full JCK 6b test suite. See 4.3 Running Tests for detailed description of using the JavaTest harness GUI to run tests.

See the JavaTest Harness User's Guide: Graphical User Interface included with this release or the JavaTest harness online help for detailed instructions about configuring and running tests.

4.3 Running Tests

You can run the JCK test suites either from the JavaTest harness GUI or from the command-line interface. If you are running interactive tests you must use the GUI. If you want to run the tests non-interactively, for example as part of a product build process, use the JavaTest harness command-line interface.

Using the JavaTest Harness GUI

  1. Complete special test setup requirements.

    See 4.5 Special Setup Instructions.

  2. Start the JavaTest harness.

    See 4.1 Starting the JavaTest Harness.

  3. Choose File -> Open Quick Start wizard from the Test Manager menu bar if the harness does not open the wizard when it launches the GUI.
  4. Choose one of the following options in the wizard and complete the associated questions:

    You are prompted to select a test suite.

    1. Start a New Test Run.
    2. Resume Work on a Test Run.

      See 4.4 Configuration Interview for a description of the structure of each interview provided by each JCK test suite.

      See the JavaTest Harness User's Guide: Graphical User Interface included with this release or the JavaTest harness online help for detailed instructions about using the configuration editor window.

  5. Click the Start Running Tests button on the Test Manager tool bar or choose Run Tests -> Start from the Test Manager menu bar.

    See the JavaTest Harness User's Guide: Graphical User Interface included with this release or the JavaTest harness online help for detailed instructions about using the Test Manager to run tests.

4.3.1 Using the Command-Line Interface

To run tests from the command line, you must use a configuration file created by the Configuration Editor. The JavaTest harness command-line interface provides a number of commands that you can use to specify individual configuration values used to run tests. However, values set in the command line override but do not change the values in the configuration file. You must use the Configuration Editor to make permanent changes in the configuration file.

See the JavaTest Harness User's Guide: Command Line Interface included with this release or the JavaTest harness online help for the commands and detailed instructions required to run tests from the command-line interface.

4.3.2 Setting Configuration Values in the Command Line

If you intend to set one or more configuration values in the command line for a test run, you must use one of the following sources to identify the appropriate configuration question-tag name:

  • The Question Log file (questionLognnnn.html) generated by the Configuration Editor in workdirectory/jtData.

  • The config.html report generated from the JavaTest harness GUI.

  • The Question Tags displayed in the Configuration Editor.

See the JavaTest Harness User's Guide: Graphical User Interface or the JavaTest harness online help for instructions required display the Question Tags in the Configuration Editor or generate reports containing the config.html file.


Note - The questionLognnnn.html and config.html files can be viewed in any web browser.


4.4 Configuration Interview

The JCK 6b test suites include configuration interviews used by the JavaTest harness to collect information from you about your test environment and to create the configurations used to run tests. The configuration interviews use the More Info panel of the Configuration Editor to provide specific configuration information based on your answers to previous interview questions. See the JavaTest Harness User's Guide: Graphical User Interface included with this release or the JavaTest harness online help for detailed instructions about using the Configuration Editor.

A configuration interview consists of the following sequence of question categories:

  1. Configuration Modes and Tests to Run Questions — The first section of each interview collects information about the mode in which you want to run the configuration interview and about which tests to run. See 4.4.1 Configuration Modes and 4.4.1.2 Specify Tests to Run for additional information.

  2. Test Environment Questions — This section of an interview collects information about the test environment used to run the JCK 6b test suite (such as the type of environment and the type of agent used). See 4.4.2.1 JCK Runtime Test Suite, 4.4.2.2 JCK Compiler Test Suite, 4.4.2.3 JCK Devtools Test Suite, and 4.4.2.4 Using Agents for additional information.


    Note - The test environment related questions are displayed only if you run the tests and the interview in Advanced mode. See 4.4.1 Configuration Modes for more information on Advanced mode.


  3. Test-Specific Questions — This section of each interview collects information about the tests from the test suite that is to be run. See Chapter 5 for detailed information.

  4. Test Run Questions — The last section of each interview collects information about how the tests are run (such as Keywords and the exclude list) and contains test suite specific information and values. See 4.4.4.2 Keywords and 4.4.4.1 Exclude Lists for additional information.

4.4.1 Configuration Modes

You can run each configuration interview in one of the following modes:

  • Advanced — Advanced mode is designed for use by expert users. This mode provides users with access to all configuration features including advanced features. In this mode the interview may contain over a hundred questions.

  • Simple — Simple mode is designed for use by non-expert users. The number of interview questions is significantly reduced due to limited usage of advanced features and extensive usage of default values. In Simple mode it is assumed that the tests are run as follows:

    • locally that is on the same machine as the interview

    • without test agents

    • each test is started in a separate virtual machine (VM)

    If these limitations are not acceptable for a particular test run, then Advanced mode must be used. The number of interview questions is also reduced when only selected tests need to be run. In this case the interview questions that are not relevant to the selected tests are not displayed.

    In Simple mode some of the default values are set based on the Sun reference implementation (RI) parameters. For example, in Simple mode the question about the syntax of the VM classpath option is not asked assuming that the syntax is the same as in the Sun RI. See Table 4-2 for more details. If the VM under test uses a different option syntax, then you can do one of the following:

    • use a wrapper script to start the VM under test that emulates the command line interface of the Sun RI

    • use Advanced mode that asks a set of questions about the syntax of options of the VM under test


Tip - In Simple mode, the Test Run Questions section of the configuration is hidden. You can access these values via the JavaTest GUI using the Quick Set Mode interview.



Tip - If you are in doubt which mode is appropriate for you, try Simple mode first. You can later switch the interview to Advanced mode and redefine standard values.



Caution - Before switching the interview from Advanced mode to Simple mode, save your configuration options because in Simple mode a number of hidden configuration settings will be initialized with default values and the answers that you already provided in Advanced mode may be lost.


Both Advanced and Simple modes can be used to certify a Java SE 6 implementation as described in Chapter 2, Procedure for Java SE 6 Technology Certification.

In the interview, configure the mode to run the tests by providing the answers from Table 4-1.

Table 4-1 Interview Answers That Configure Configuration Mode

Question

Answer

Advanced Mode — Do you wish to use the Advanced mode?

Yes — To run the interview in Advanced mode

No — To run the interview in Simple mode

4.4.1.1 Simple Mode Default Values

In Simple mode a number of interview questions are hidden because they use answers initialized with default values.

Table 4-2 lists the default values that are set in Simple mode for the JCK Runtime test suite. The questions are listed in the order in which they appear in Advanced mode.

Table 4-2 Interview Answers — Default Values in Simple Mode for JCK Runtime Test Suite

Question

Default Value

Comments

Configuration Name — Provide a short identifier that names the configuration you are creating.

jck_${product}_${OS}

Simple mode assumption.

Description — Provide a short description that identifies the configuration you are creating.

None

Simple mode assumption.

Native Code — Does your system support direct access to native code?

Yes

Simple mode assumption.

JVM TI Support — Is the Java virtual machine Tool Interface (JVM TI) implementation provided?

Yes

Simple mode assumption.

Fallback Support — Have you configured the product VM in this test run to use the fallback mode (the type inference strategy) of class verification when the typechecking strategy fails?

Yes

Simple mode assumption.

Fall back to the older verification when typechecking fails.

Headless System — Is the system you are testing headless?

No

Simple mode assumption.

Display is available.

Test Execution Mode — Select the test execution mode

MultiJVM

Simple mode assumption.

Testing Locally — Are you running these tests on the same system used for this interview?

Yes

Simple mode assumption.

Testing locally. No JavaTest agent is used.

Runtime Classpath — For the Java launcher that runs the tests, is the classpath specified with an environment variable or a command-line option?

command line option

Simple mode assumption.

Classpath Option — Enter an option template for use in setting the classpath for the Java launcher that runs the tests.

-classpath #

The same parameter as in the Sun RI.

Location of native libraries — Are all of the native libraries in one location?

Yes

Simple mode assumption.

Native Library Path Value — What is the name of the directory that contains the libraries of native code?

PATH (for Windows)

or LD_LIBRARY_PATH (for Unix, Linux, and other)

The same parameter as in the Sun RI.

JVM TI Agent Option String — Enter an option template for use in specifying the JVM TI agent library. Use the character `#' in the option template to indicate where the library name is specified. Use the character `@' to indicate where the agent options string is specified.

-agentlib:#=@

The same parameter as in the Sun RI.

JVM TI Agent start-up in Live phase — Is JVM TI Agent start-up in Live phase supported?

Yes

Simple mode assumption.

JVM TI AttachConnector implementation — Specify the fully-qualified class name used for starting native agent in the VM during the live phase.

javasoft.sqe.jck.lib.attach.

JVMTIAttachConnector

Other assumptions.

JAX-WS Version— What version of JAX-WS technology does your implementation support.

JAX-WS 2.1

Std Activation Port — Use the standard RMI activation port for the RMI daemon?

Yes

The same parameter as in the Sun RI.

System Property Specification — Does your platform use the standard method of specifying system properties on the command line?

–Dname=value

The same parameter as in the Sun RI.

DISPLAY Variable — Some tests might require that you set the "DISPLAY" environment variable. Enter a value for the DISPLAY variable for the system on which you run the tests.

DISPLAY OS environment or localhost:0.0

The same parameter as in the Sun RI.

Verifier Option — JCK requires that the verifier is fully enabled for each test that is run. Specify all options that might be required by your Java launcher to enable the verifier.

–Xfuture

The same parameter as in the Sun RI.

Can Play Audio — Does your system provide support for producing audio?

Yes

Other assumptions.

Use the Provided Audio File — Does your system support WAV, AU or SND audio formats for playback?

Yes

Other assumptions.

.WAV

Security Authorization Policy — Which of the following mechanisms does your product use to specify the Java Security Authorization Policy?

standard system property

The same parameter as in the Sun RI.

Security Login Configuration — Which of the following mechanisms does your product use to specify the Java Security Login Configuration?

standard system property

The same parameter as in the Sun RI.

Expect OutOfMemoryError — Can OutOfMemoryError be thrown on the system under test when memory is allocated?

Yes

Simple mode assumption.

Restrict TCP Ports — Does your operating system restrict the set of TCP ports available to tests that request a specific port?

No

Simple mode assumption.

Local Host Name — What is the host name of the system on which you run the tests?

localhost

Other assumptions.

Local Host IP Address — What is the IP address of the system on which you run the tests?

127.0.0.1

Other assumptions.

Restrict UDP Ports — Does your operating system restrict the set of UDP ports available to tests that request a specific port?

No

Simple mode assumption.

Need Proxy — Do you use a proxy to access the HTTP and FTP URLs you have just entered?

No

Simple mode assumption.

ORB Host — What is the name of the machine on which the Java IDL name server is run?

localhost

Other assumptions.

ORB Port — What is the port on which the Transient Name Service listens?

900

Other assumptions.

Printer — Is your system connected to a printer that is online?

Yes

Simple mode assumption.

AWT Robot — Does your system provide support for automated event generation, as defined by the java.awt.robot classes?

Yes

Simple mode assumption.

JDWP Support— Is the Java Debugger Wire Protocol (JDWP) implementation provided?

Yes

Simple mode assumption.

Transport Class Name — Specify the fully-qualified class name used for establishing connection between the tests debuggers and JDWP implementation.

javasoft.sqe.jck.lib.jpda.jdwp.

SocketTransportService

Other assumptions.

Attaching or Listening debugger — Should the debugger listen for connections from or attach to the JDWP implementation?

attaching

Other assumptions.

Connection Address for Debugger — Specify the address for the debugger to use when listening for connections from or attaching to the JDWP implementation.

Passive agent host (${RuntimeRemoteAgentInterview.

passiveHost})

or localhost as host and 35000 as port

Other assumptions.

VM Suspended On Start — Does the specified JDWP implementation configuration provide that the Java runtime is suspended on-start?

Yes

Simple mode assumption.

JDWP Command-line Options — Specify the command-line options for activating JDWP.

-agentlib:jdwp=server=y,

transport=dt_socket,

address=${RuntimeJDWPInterview.

transportAddress},suspend=Y

The same parameter as in the Sun RI.

RowSet Factory — Enter the name of the factory class for creating the rowset implementations.

javasoft.sqe.tests.api.javax.sql.

rowset.impl.RowSetFactoryImpl

Other assumptions.

JPLIS Agent start-up in Live phase — Does your runtime supports instrumentation agent start-up after the JVM is launched?

Yes

Simple mode assumption.

JPLIS AttachConnector implementation — Specify the fully-qualified class name used for starting instrumentation agent sometime after the JVM is launched.

javasoft.sqe.jck.lib.attach.

JPLISAttachConnector

Other assumptions.

4.4.1.2 Specify Tests to Run

You can specify the tests that are run either in the command line or in the Configuration Editor. If you specify a test folder, all tests located hierarchically beneath it are automatically selected for you. See the JavaTest Harness User's Guide: Graphical User Interface, the JavaTest Harness User's Guide: Command-Line Interface, or the JavaTest harness online help for a detailed description of how to specify the tests that are run.

In the interview, select the tests to run by providing the answers from Table 4-3.

Table 4-3 Interview Answers That Configure The Tests to Run

Question

Answer

Specify Tests to Run — Do you wish to run only selected sections of the test suite?

Yes — To specify the tests to run

No — To run all tests

4.4.2 Test Environment Questions


Note - The configuration interview displays test environment related questions only if you run the tests and the interview in Advanced mode. See 4.4.1 Configuration Modes for more information on Advanced mode.


The JCK 6b test suites can be run in many different test environments. The first section of each interview contains a series of sub-interviews that collect the information required to configure a test environment.

To successfully pass the tests in the JCK 6b test suites, you must set the following options in the Test Execution section of each interview:

  • Use the Verifier Option question to enable full class file verification in the configuration used to run the tests.

    To enable full class file verification, set the -Xfuture option in the Verifier Option field of the JCK interview.

  • When you select Yes in the Test Platform MultiJVM question, use the Other Options question to specify the extension directory that extends the tested platform for testing the pluggability of locales provided by an implementation.


    Note - To test the pluggability of locales provided by an implementation, a tested platform must be extended by the JAR files in jck/lib/extensions each time it is started.


    For Sun implementations, set the following option in the Other Options field:

    -Djava.ext.dirs=jck/lib/extensions: java-home/lib/ext:jre-home/lib/ext

    If you do not select Yes in the Test Platform MultiJVM question you cannot use the Other Options question to specify the extension directory. When you start an agent (from a command line or a browser) to run tests, specify jck/lib/extensions as an additional extension directory.

4.4.2.1 JCK Runtime Test Suite

The interview presents a series of questions and uses their answers to create a configuration for the test environment of the JCK 6b runtime test suite. The following sections describe the test environments that might be configured for the test run:


Note - The illustrations and descriptions in the following sections describe the interactions of the various VMs and systems involved in running the tests. In addition, some tests require the use of other servers, possibly on additional systems. For simplicity these servers and systems are not included here. These remote systems are described in 4.4.2.1 JCK Runtime Test Suite and in Chapter 5.


Multiple VMs Without an Agent

In the interview, when you provide the answers from Table 4-4 or from Table 4-5, the Configuration Editor creates a configuration that uses multiple VMs without an agent to run the runtime tests.

Table 4-4 Interview Answers (Runtime Running Locally) That Configure Multiple VMs Without an Agent

Question

Answer

Test Execution Mode — Select the test execution mode

MultiJVM

Testing Locally — Are you running these tests on the same system used for this interview?

Yes

Table 4-5 Interview Answers (Runtime Running Remotely) That Configure Multiple VMs Without an Agent

Question

Answer

Test Execution Mode — Select the test execution mode

MultiJVM

Testing Locally — Are you running these tests on the same system used for this interview?

No

Test Platform — Which operating system is used on the test platform?

Any choice but Other

In the test environment illustrated in Figure 4-1, the JavaTest harness and the JCK tests run in separate Java technology runtimes. The harness starts a new VM for every test. This test environment does not use an agent.

Figure 4-1 Runtime Test Environment - Multiple VMs Without an Agent
Multiple VMs Without an Agent

Note - If the harness displays errors indicating that classes are not found, check that the classpaths were set correctly when configuring the classpath for each of the VMs.


Multiple VMs With an Agent

In the interview, when you provide the answers to questions listed in Table 4-6, the Configuration Editor creates a configuration that uses multiple VMs with an agent for running the runtime tests.

Table 4-6 Interview Answers That Configure Multiple VMs With an Agent

Question

Answer

Test Execution Mode — Select the test execution mode

MultiJVM

Testing Locally — Are you running these tests on the same system used for this interview?

No

Test Platform — Which operating system is used on the test platform?

Other

In the test environment illustrated in Figure 4-2, the JavaTest harness runs on a separate computer (using one of the reference VMs) while an agent runs the tests in separate VMs on the system under test. The agent, JCK tests, and the runtime product run in separate Java runtime environments on the target system. The harness starts a new VM for every test. See 4.4.2.4 Using Agents for a description of the types of agents that you can use.

Figure 4-2 Runtime Test Environment — Multiple VMs With an Agent
Multiple VMs With an Agent

Note - If the harness displays errors indicating that classes are not found, check that the class paths were set correctly when configuring the classpath for each of the VMs.


Multiple VMs Group Without an Agent
Table 4-7 Interview Answers That Configure Multiple VMs Group Without an Agent

Question

Answer

Test Execution Mode — Select the test execution mode

MultiJVM group

Testing Locally — Are you running these tests on the same system used for this interview?

Yes

MutliJVM group mode is intended to reduce the overhead of starting VMs by sharing VM instances among groups of test executions. In this mode the harness starts an internal agent VM on the system that JavaTest is run on and passes a group of the test execution to this agent. Once the internal agent completes the test group run, it closes. The new internal agent VM is started for the next group of tests. The JavaTest harness starts socket server in particular port to communicate with internal agents running on the same system. By default, the port number is chosen from the list of available ports but you can override this parameter.

Figure 4-3 Runtime Test Environment — Multiple VMs Group Without an Agent
Multiple VMs Group Without an Agent

Note - If you have a firewall installed on your system, make sure it allows incoming connections from the local host.


The internal agent waits for a test execution request from the JavaTest harness. If no test request received during a timeout, the agent VM is automatically closed.

The configuration of the MutliJVM group test execution is the same as the configuration of the MutliJVM mode except for several parameters listed in Table 4-8 that control the MutliJVM group mode execution.


Note - These parameters are not visible in the JCK configuration interview and can only be changed by setting corresponding test environments variables via the JavaTest harness command line interface. For example:

javatest -jar lib/javatest.jar -Ejck.env.runtime.testExecute.groupMode.serverPort=8080

This variable sets TCP port used for listening for incoming connections from internal agents.


See Additional Setup Commands chapter in the JavaTest Harness Command-Line Interface User's Guide.

Table 4-8 Environment variables that control execution of the MutliJVM group mode

Variable

Description

jck.env.runtime.testExecute.groupMode.groupSize

Controls a number of tests executed as a group in the same agent VM instance started by the JavaTest harness. Setting groupSize value to >1 turns the group mode on. Setting groupSize to 1 turns the group mode off. Setting groupSize value to 0 removes limitation on the number of tests executed by one VM (Default is 100). Default group size provides optimal balance between performance and stability of JCK runs.

jck.env.runtime.testExecute.groupMode.serverPort

A TCP port used for listening for incoming connections from internal agents. Default value is 0, which means that JVM will automatically choose a free port on the test system. You can specify a particular port number to avoid firewall issues.

jck.env.runtime.testExecute.groupMode.timeout

Timeout (in seconds) between test executions during which an agent is active. Default is 60 seconds. Increase this value if necessary.

jck.env.runtime.testExecute.groupMode.agentArgs

Additional agent command line arguments passed to internal agents while starting. For example, you can specify "-trace" option to enable diagnostic output from the agent.

jck.env.runtime.testExecute.groupMode.closeIfFail

A policy used to close agents after test execution if a test status is fail. Default is false, i.e. unset.

jck.env.runtime.testExecute.groupMode.keepOnError

A policy used to close agents after test execution if a test status is error. Default is false, i.e. unset.

jck.env.runtime.testExecute.groupMode.debug

Setting this variable enables additional logging. By default debug is off.


Note - When you select the MutliJVM group mode, JCK does its best to optimize execution by grouping tests and executing them in the same VM instance. However, in case tests cannot be grouped (for example, tests marked by the following keywords: "interactive", "jdwp", "jplis", "jvmti", "jplislivephase", "jvmtilivephase", "jniinvocationapi", "only_once"), the tests will be executed individually.


Multiple VMs Group With an Agent
Table 4-9 Interview answers that configure Multiple VMs group with an agent

Question

Answer

Test Execution Mode — Select the test execution mode

MultiJVM group

Testing Locally — Are you running these tests on the same system used for this interview?

No

Test Platform

Other

When you answer interview questions as specified in Table 4-9, the JavaTest harness runs on a separate computer in the test environment (using one of the reference VMs) while an agent runs tests in separate VMs system under test. The agent, JCK tests, and the runtime product run in separate Java runtime environments on the target system. The agent starts a new internal agent VM on the target system and passes a group of test execution to this agent.

Figure 4-4 Runtime Test Environment — Multiple VMs Group With an Agent
Multiple VMs Group With an Agent

All MultiJVM group mode configuration parameters, mentioned in Table 4-8 apply to this execution mode.

See 4.4.2.4 Using Agents for a description of the types of agents that you can use.

Single VM With an Agent

In the interview, when you select SingleJVM in the following question, the Configuration Editor creates a configuration using a single VM with an agent to run the runtime tests:

Test Execution Mode — Select the test execution mode

In the test environment illustrated in Figure 4-5, the JavaTest harness runs on a separate computer (using one of the Sun reference VMs), while an agent and the JCK tests run on your runtime product located on the target system.

This test environment is used when the system under test does not support multiple processes. Because the agent and the JCK tests run in the same VM, the classpath for the agent VM must be correct for both the agent and the tests. See 4.4.2.4 Using Agents for a description of the types of agents that you can use.


Note - If the harness displays errors indicating that classes are not found, check that the class paths were set correctly when configuring the class path for each of the VMs.


Figure 4-5 Runtime Test Environment — Single VM With an Agent
Single VM With an Agent

If your product runs in a browser on Win32 or Solaris platform, Figure 4-6 can also represent your test configuration. The dashed box in Figure 4-6 represents the browser. In the test environment illustrated in Figure 4-6, the JavaTest harness, agent and the tests run in a browser on the same system (the runtime product).

Figure 4-6 Runtime Test Environment — Single VM Browser Environment
Single VM Browser Environment

A significant difference exists between the single and multiple VM environment configurations. In multiple VM environment configuration, all of the information required to configure and execute the Java runtime environment under test is specified in the JCK configuration file. In single VM environments, configuration information is spread across the following multiple locations:

  • In the current JCK configuration

  • On the command line for the Java runtime environment under test

  • In the HTML file used to load the agent applet (if you are testing a browser)

Because the Java runtime environment under test is only launched once for a test run, the bulk of the JCK configuration happens when the VM is started.

  • The Java technology security manager must be correctly configured to run tests.

  • The native code library path must be configured to allow the Java Native Interface (JNITM) implementation and Extra-Attribute tests to run.

  • Any necessary VM options must be specified.

  • Any other required shell environment variables must be specified before the runtime under test is started with the agent.

  • If you are not testing a browser and not using the -loadDir option, the class path must also be correctly specified to include the agent and the test classes before starting the runtime under test.

  • If you are testing a browser implementation of the Java runtime environment, you must supply the correct classpath in an HTML file.

4.4.2.2 JCK Compiler Test Suite

The interview presents a series of questions and uses their answers to create a configuration for the test environment of the JCK 6b compiler test suite. The following sections describe the possible configurations of test environments used to run the tests of the JCK compiler test suite:

It is important to note how the JCK compiler tests work. RMI compilers and the Java programming language compiler (Java compiler) compile test files and the resulting class files are executed on a Sun reference runtime implementation. To compile and execute compiler tests, you must configure five test commands that invoke the following compilers and runtime:

  • Java complier

  • RMI compiler

  • RMI Compiler v1.1

  • RMI Compiler v1.2

  • Sun reference runtime

Table 4-10 describes the different types of compiler tests.

Table 4-10 Types of JCK Compiler Tests

Type of Test

Processes Started

Negative

Only the Java compiler is started, the compiler is expected to fail and produce an error. A new compiler is started for every test.

Positive (Java compiler)

A new Java compiler is started for every test. Output of the compiler is always checked using a new reference VM.

Positive (RMI compiler)

New Java and RMI compilers are started for every test. Output of the compilers is always checked using a new reference VM.

Multiple VMs Without an Agent

In the interview, when you provide the answers from Table 4-11, the Configuration Editor creates a configuration that uses multiple VMs without an agent to run the compiler tests.

Table 4-11 Interview Answers (Compiler Running Locally) That Configure Multiple VMs Without an Agent

Question

Answer

Test Execution Mode — Select the test execution mode

MultiJVM

Testing Locally — Are you running these tests on the same system used for this interview?

Yes

In the test environment illustrated in Figure 4-7, the JavaTest harness, the compilers under test, and the reference VMs that test the compiler output are run on the same computer but in separate processes. This test environment does not use an agent. Processes are started as required depending on the type of compiler test.

Figure 4-7 Compiler Test Environment — Multiple VMs Without an Agent
Multiple VMs Without an Agent

Note - If the harness displays errors indicating that classes are not found, check that the classpaths were set correctly when configuring the classpath for each of the VMs.


Multiple VMs With an Agent

In the interview, when you provide the answers from Table 4-12, the Configuration Editor creates a configuration that uses multiple VMs with an agent to run the compiler tests.

Table 4-12 Interview Answers (Compiler Running Remotely) That Configure Multiple VMs With an Agent

Question

Answer

Test Execution Mode — Select the test execution mode

MultiJVM

Testing Locally — Are you running these tests on the same system used for this interview?

No

Test Platform — Which operating system is used on the test platform?

Any choice

In the test environment illustrated in Figure 4-8, the JavaTest harness and the reference VMs that test the compiler output run on the same computer. The agent, the Java compiler and the RMI compiler under test run on another system (the system under test) in separate processes. Processes are started as required depending on the type of compiler test. See Table 4-10 for a description of the different types of compiler tests. See 4.4.2.4 Using Agents for a description of the types of agents that you can use.

Figure 4-8 Compiler Test Environment — Multiple VMs With an Agent
Multiple VMs With an Agent

Note - If the harness displays errors indicating that classes are not found, check that the classpaths were set correctly when configuring the classpath for each of the VMs.


Multiple VMs Group Without an Agent
Table 4-13 Interview answers that configure Multiple VMs group without an agent

Question

Answer

Test Execution Mode — Select the test execution mode

MultiJVM group

Testing Locally — Are you running these tests on the same system used for this interview?

Yes

MultiJVM group mode is intended to reduce the overhead of starting VMs by sharing Java complier and Sun reference runtime VM instances among groups of test executions. In this mode the harness starts an internal agent VM on the system that JavaTest is run on and passes a group of the test execution to this agent. Once the internal agent completes its test group run, it closes. The new internal agent VM is started for the next group of tests. The JavaTest harness starts socket server in particular port to communicate with internal agents running on the same system. By default, the port number is chosen from the list of available ports but you can override this parameter.

Figure 4-9 Compiler Test Environment — Multiple VMs Group Without an Agent
Multiple VMs Group Without an Agent

Note - If you have a firewall installed on your system, make sure it allows incoming connections from the local host.


The internal agent waits for a test execution request from the JavaTest harness. If no test request received during a timeout, the agent VM is automatically closed.


Note - The MultiJVM group is not applied to RMI compiler. It is applied only to Java compiler and Sun reference runtime.


The MultiJVM group mode expects Java compiler under test to be configured to run via the Java Compiler API interface. The configuration of the MultiJVM group mode is the same as the configuration of the MultiJVM mode except for several parameters listed in Table 4-14 that control the MultiJVM group mode execution.


Note - These parameters are not visible in the JCK configuration interview and can only be changed by setting corresponding test environments variables via the JavaTest harness command line interface. For example:

javatest -jar lib/javatest.jar -Ejck.env.compiler.testCompile.groupMode.serverPort=8080

This variable sets TCP port used for listening for incoming connections from internal agents.


See Additional Setup Commands chapter in the JavaTest Harness Command-Line Interface User's Guide.

Table 4-14 Environment variables that control execution of the MutliJVM group mode

Variable

Description

jck.env.compiler.testCompile.groupMode.groupSize

Controls a number of tests executed as a group in the same agent VM instance started by the JavaTest harness. Setting groupSize value to >1 turns the group mode on. Setting groupSize to 1 turns the group mode off. Setting groupSize value to 0 removes limitation on the number of tests executed by one VM (Default is 100). Default group size provides optimal balance between performance and stability of JCK runs.

jck.env.compiler.testCompile.groupMode.serverPort

A TCP port used for listening for incoming connections from internal agents. Default value is 0, which means that JVM will automatically choose a free port on the test system. You can specify a particular port number to avoid firewall issues.

jck.env.compiler.testCompile.groupMode.timeout

Timeout (in seconds) between test executions during which the agent is active. Default is 60 seconds. Increase this value if necessary.

jck.env.compiler.testCompile.groupMode.agentArgs

Additional agent command line arguments passed to internal agents while starting. For example, you can specify "-trace" option to enable diagnostic output from the agent.

jck.env.compiler.testCompile.groupMode.closeIfFail

A policy used to close agents after test execution if a test status is fail. Default is false, i.e. unset.

jck.env.compiler.testCompile.groupMode.keepOnError

A policy used to close agents after test execution if a test status is error. Default is false, i.e. unset.

jck.env.compiler.testCompile.groupMode.debug

Setting this variable enables additional logging. By default debug is off.

Table 4-15 Environment variables that control execution of the MutliJVM group mode for the Sun's reference runtime

Variable

Description

jck.env.compiler.compRefExecute.groupMode.groupSize

Controls a number of tests executed as a group in the same agent VM instance started by the JavaTest harness. Setting groupSize value to >1 turns the group mode on. Setting groupSize to 1 turns the group mode off. Setting groupSize value to 0 removes limitation on the number of tests executed by one VM (Default is 100). Default group size provides optimal balance between performance and stability of JCK runs.

jck.env.compiler.compRefExecute.groupMode.serverPort

A TCP port used for listening for incoming connections from internal agents. Default value is 0, which means that JVM will automatically choose a free port on the test system. You can specify a particular port number to avoid firewall issues.

jck.env.compiler.compRefExecute.groupMode.timeout

Timeout (in seconds) between test executions during which the agent is active. Default is 60 seconds. Increase this value if necessary.

jck.env.compiler.compRefExecute.groupMode.agentArgs

Additional agent command line arguments passed to internal agents while starting. For example, you can specify "-trace" option to enable diagnostic output from the agent.

jck.env.compiler.compRefExecute.groupMode.closeIfFail

A policy used to close agents after test execution if a test status is fail. Default is false, i.e. unset.

jck.env.compiler.compRefExecute.groupMode.keepOnError

A policy used to close agents after test execution if a test status is error. Default is false, i.e. unset.

jck.env.compiler.compRefExecute.groupMode.debug

Setting this variable enables additional logging. By default debug is off.


Note - When you select the MutliJVM group mode, JCK does its best to optimize execution by grouping tests and executing them in the same VM instance. However, in case tests cannot be grouped (for example, tests marked by the keyword "only_once"), the tests will be executed individually.


Multiple VMs Group With an Agent
Table 4-16 Interview answers that configure Multiple VMs group with an agent

Question

Answer

Test Execution Mode — Select the test execution mode

MultiJVM group

Testing Locally — Are you running these tests on the same system used for this interview?

No

Test Platform

Other

When you answer interview questions as specified in Table 4-16, the JavaTest harness runs on a separate computer in the test environment (using one of the reference VMs) while an agent runs tests in separate VMs system under test. The agent, JCK tests, and the compiler run in separate Java runtime environments on the target system. The agent starts a new internal agent VM on the target system and passes a group of test execution to this agent. However, the reference runtime is started on the system that JavaTest is run.

Figure 4-10 Compiler Test Environment — Multiple VMs Group With an Agent
Multiple VMs Group With an Agent

All Multi JVM group mode configuration parameters, mentioned in Table 4-14 apply to this execution mode. See 4.4.2.4 Using Agents for a description of the types of agents that you can use.

Single VM With an Agent

In the interview, when you select SingleJVM in the following question, the Configuration Editor creates a configuration that uses a single VM with an agent to run the compiler tests:

Test Execution Mode — Select the test execution mode

In the test environment illustrated in Figure 4-11, the JavaTest harness and the reference VM that tests the compiler output are run on the same computer. An agent, the Java compiler, and the RMI compiler run in the same process on another system (the system under test). This test configuration is used when the test system does not support multiple processes.

The configuration for single VM test commands that use an agent to invoke a compiler is different from the multiple VM commands. To achieve a better performance, the command, executing reference runtime, is the same as in the Multi JVM group mode in which the harness starts an internal agent VM on the system that JavaTest is run on and passes a group of the test execution to this agent. You can change this to use Multi JVM mode by setting the environment variable jck.env.compiler.compRefExecute.groupMode.groupSize to 1. See Table 4-15 Environment variables that control execution of the MutliJVM group mode for the Sun's reference runtime for more information.

Because the agent and the compilers run in the same VM the classpath for the agent VM must be correct for the agent and the compilers. See 4.4.2.4 Using Agents for a description of the types of agents that you can use.

Figure 4-11 Compiler Test Environment — Single VM With an Agent
Single VM With an Agent

Note - If the harness displays errors indicating that classes are not found, check that the classpaths were set correctly when configuring the class path for each of the VMs.


4.4.2.3 JCK Devtools Test Suite

The interview presents a series of questions and uses their answers to create a configuration for the test environment of the JCK devtools (devtools) test suite. The following sections describe the possible configurations of test environments used to run the tests of the devtools test suite:

It is important to note the following descriptions of how the devtools tests work:

  • The schema compiler compiles test files and executes the resulting class files on a Sun reference runtime implementation.

  • The schema generator generates schemas from test files and validates the resulting schemas on the Sun reference runtime implementation.

Both the schema compiler and the schema generator are types of compilers. The schema compiler maps from schemas to Java class files. The schema generator maps from Java class files to schemas.


Note - Devtools compiler is used in this section to refer to both the schema compiler and the schema generator.


To compile and execute devtools compiler tests, you must configure three test commands that invoke the following:

  • Schema compiler

  • Schema generator

  • Sun reference runtime

Table 4-17 describes the different types of devtools compiler tests.

Table 4-17 Types of Devtools Compiler Tests

Type of Test

Processes Started

Negative (Schema Compiler)

Only the Schema compiler is started, the compiler is expected to fail and produce an error. A new compiler is started for every test.

Positive (Schema Compiler)

A new Schema compiler is started for every test. Output of the compiler is always checked using a new reference VM.

Positive (Schema Generator)

A new Schema generator is started for every test. Output of the generator is always checked using a new reference VM.

Multiple VMs Without an Agent

In the interview, when you provide the answers from Table 4-11, the Configuration Editor creates a configuration that uses multiple VMs without an agent to run the devtools compiler tests.

Table 4-18 Interview Answers (Devtools Compiler Running Locally) That Configure Multiple VMs Without an Agent

Question

Answer

Test Execution Mode — Select the test execution mode

MultiJVM

Testing Locally — Are you running these tests on the same system used for this interview?

Yes

In the test environment illustrated in Figure 4-12, the JavaTest harness, the devtools compiler under test, and the reference VMs that test the devtools compiler output are run on the same computer but in separate processes. This test environment does not use an agent. Processes are started as required depending on the type of devtools compiler test.

Figure 4-12 Devtools Compiler Test Environment — Multiple VMs Without an Agent
Multiple VMs Without an Agent

Note - If the harness displays errors indicating that classes are not found, check that the classpaths were set correctly when configuring the classpath for each of the VMs.


Multiple VMs With an Agent

In the interview, when you provide the answers from Table 4-12, the Configuration Editor creates a configuration that uses multiple VMs with an agent to run the devtools compiler tests.

Table 4-19 Interview Answers (Devtools Compiler Running Remotely) That Configure Multiple VMs With an Agent

Question

Answer

Test Execution Mode — Select the test execution mode

MultiJVM

Testing Locally — Are you running these tests on the same system used for this interview?

No

In the test environment illustrated in Figure 4-13, the JavaTest harness and the reference VMs that test the devtools compiler output run on the same computer. The agent and the devtools compilers run on another system (the system under test) and in separate processes. Processes are started as required depending on the type of compiler test. See Table 4-17 for a description of the types of compiler tests. See 4.4.2.4 Using Agents for a description of the types of agents that you can use.

Figure 4-13 Devtools Compiler Test Environment — Multiple VMs With an Agent
Multiple VMs With an Agent

Note - If the harness displays errors indicating that classes are not found, check that the classpaths were set correctly when configuring the classpath for each of the VMs.


Multiple VMs Group Without an Agent
Table 4-20 Interview answers (Devtools Compiler Running Locally) that configure Multiple VMs group without an agent

Question

Answer

Test Execution Mode — Select the test execution mode

MultiJVM group

Testing Locally — Are you running these tests on the same system used for this interview?

Yes

MultiJVM group mode is intended to reduce the overhead of starting VMs by sharing Sun reference complier and Sun reference runtime VM instances among groups of devtools test executions. In this mode the harness starts an internal agent VM on the system that JavaTest is run on and passes a group of the test execution to this agent. Once the internal agent completes its test group run, it closes. The new internal agent VM is started for the next group of tests. The JavaTest harness starts socket server in particular port to communicate with internal agents running on the same system. By default, the port number is chosen from the list of available ports but you can override this parameter.

Figure 4-14 Devtools Compiler Test Environment — Multiple VMs Group Without an Agent
Multiple VMs Group Without an Agent

Note - If you have a firewall installed on your system, make sure it allows incoming connections from the local host.


The internal agent waits for a test execution request from the JavaTest harness. If no test request received during a timeout, the agent VM is automatically closed.


Note - The MultiJVM group is not applied to Devtools compilers. It is applied only to Sun reference runtime and Sun reference Java compiler.



Note - In this mode, the interview question about the path to Sun reference java compiler will not be asked. The reference compiler will be automatically configured to be run via Java Compiler API interface using Sun reference runtime.


The configuration of the MultiJVM group mode is the same as the configuration of the MultiJVM mode except for several parameters listed in Table 4-14 that control the MultiJVM group mode execution.


Note - These parameters are not visible in the JCK configuration interview and can only be changed by setting corresponding test environments variables via the JavaTest harness command line interface. For example:

javatest -jar lib/javatest.jar -Ejck.env.devtools.testExecute.groupMode.groupSize=400 This variable sets the number of tests grouped in one VM instance.

See Additional Setup Commands chapter in the JavaTest Harness Command-Line Interface User's Guide.


Table 4-21 Environment variables that control execution of the MultiJVM group mode

Variable

Description

jck.env.devtools.testExecute.groupMode.groupSize

Controls a number of tests executed as a group in the same agent VM instance started by the JavaTest harness. Setting groupSize value to >1 turns the group mode on. Setting groupSize to 1 turns the group mode off. Setting groupSize value to 0 removes limitation on the number of tests executed by one VM (Default is 100). Default group size provides optimal balance between performance and stability of JCK runs.

jck.env.devtools.testExecute.groupMode.serverPort

A TCP port used for listening for incoming connections from internal agents. Default value is 0, which means that JVM will automatically choose a free port on the test system. You can specify a particular port number to avoid firewall issues.

jck.env.devtools.testExecute.groupMode.timeout

Timeout (in seconds) between test executions during which an agent is active. Default is 60 seconds. Increase this value if necessary.

jck.env.devtools.testExecute.groupMode.agentArgs

Additional agent command line arguments passed to internal agents while starting. For example, you can specify "-trace" option to enable diagnostic output from the agent.

jck.env.devtools.testExecute.groupMode.closeIfFail

A policy used to close agents after test execution if a test status is fail. Default is false, i.e. unset.

jck.env.devtools.testExecute.groupMode.keepOnError

A policy used to close agents after test execution if a test status is error. Default is false, i.e. unset.

jck.env.devtools.testExecute.groupMode.debug

Setting this variable enables additional logging. By default debug is off.


Note - When you select the MutliJVM group mode, JCK does its best to optimize execution by grouping tests and executing them in the same VM instance. However, in case tests cannot be grouped (for example, tests marked by the keyword "only_once"), the tests will be executed individually.


Multiple VMs Group With an Agent
Table 4-22 Interview answers (Devtools Compiler Running Remotely) that configure Multiple VMs group with an agent

Question

Answer

Test Execution Mode — Select the test execution mode

MultiJVM group

Testing Locally — Are you running these tests on the same system used for this interview?

No

When you answer interview questions as specified in Table 4-22, the JavaTest harness runs on a separate computer in the test environment (using one of the reference VMs) while an agent runs tests in separate VMs system under test. The agent, JCK tests, and the devtools compiler run in separate Java runtime environments on the target system. The agent starts a new internal agent VM on the target system and passes a group of test execution to this agent. However, the reference runtime is started on the system that JavaTest is run.

Figure 4-15 Devtools Compiler Test Environment — Multiple VMs Group With an Agent
Multiple VMs Group With an Agent

All Multi JVM group mode configuration parameters, mentioned in Table 4-21 apply to this execution mode. See 4.4.2.4 Using Agents for a description of the types of agents that you can use.

Single VM With an Agent

In the interview, when you select SingleJVM in the following question, the Configuration Editor creates a configuration that uses a single VM with an agent to run the devtools compiler tests:

Test Execution Mode — Select the test execution mode


Note - In this test environment, the harness uses an agent when running tests locally or remotely.


In the test environment illustrated in Figure 4-16, the JavaTest harness and the reference VM that tests the compiler output are run on the same computer while the agent and the compilers under test run on another system (the system under test) in the same VM. This test configuration is used when the test system does not support multiple processes.

The configuration for single VM test commands that use an agent to invoke a compiler is different from the multiple VM commands. To achieve better performance, the command, executing reference runtime and reference compiler, is the same as in the Multi JVM group mode, in which the harness starts an internal agent VM on the system that JavaTest is run on and passes a group of the test execution to this agent. You can change this to use Multi JVM mode by setting the environment variable jck.env.devtools.testExecute.groupMode to 1. See chapter Multiple VMs Group Without an Agent for more information about the group mode.

Because the agent and the compilers run in the same VM, the classpath for the agent VM must be correct for both the agent and the compilers. See 4.4.2.4 Using Agents for a description of the types of agents that you can use.

Figure 4-16 Devtools Compiler Test Environment — Single VM With an Agent
Single VM With an Agent

Note - If the harness displays errors indicating that classes are not found, check that the classpaths were set correctly when configuring the classpath for each of the VMs.


4.4.2.4 Using Agents

When you configure your test environment, you specify whether you use an active or passive agent to run the tests. Active agents initiate the connection with the JavaTest harness, as opposed to passive agents that require the JavaTest harness to initiate the connection.

If you are using the JavaTest harness agent provided by the JavaTest harness, see the JavaTest Agent User's Guide or the JavaTest harness online help for detailed instructions about configuring and using JavaTest harness agents to run tests.

If you are using another agent, see the documentation provided with that agent for detailed instructions about configuring and using the agent to run tests.

4.4.3 Test-Specific Questions

The second section of each interview contains sub interviews that collect configuration information about the tests to be run. Each interview uses the More Info panel of the configuration editor to provide specific test configuration information based on your answers to previous interview questions.

For additional information about special setup and configuration requirements for running specific tests in the test suite, see Chapter 5.

4.4.4 Test Run Questions

The last section of each interview specifies how to run the tests. For example, you can run tests concurrently, set the time factor, and include or exclude tests from the test run by using keywords and using exclude lists.

See the JavaTest Harness User's Guide: Graphical User Interface included with this release or the JavaTest harness online help for detailed instructions about using the configuration editor window.


Tip - In Simple mode, theTest Run Questions section of the configuration is hidden. You can access these values via the JavaTest GUI using the Quick Set Mode interview.


4.4.4.1 Exclude Lists

In the command line or in the GUI configuration editor window, you can specify the exclude lists used when running tests. You can choose the exclude list provided by the JCK 6b test suite or you can run the tests without using an exclude list. If your system has Internet access, you can also configure the JavaTest harness to automatically check for and download updated JCK 6b exclude lists before running tests. See the JavaTest Harness User's Guide: Graphical User Interface, the JavaTest Harness User's Guide: Command-Line Interface, or the JavaTest harness online help for a detailed description of specifying one or more exclude lists when running tests.

4.4.4.2 Keywords

Each test can contain one or more keywords defined by the JCK test suite that can be used to restrict the set of tests that are run. Keywords can also be associated with different groups of tests within the test suite. In the command line or in the GUI configuration editor window, you can specify the keywords used to filter the tests that are run. See the JavaTest Harness User's Guide: Graphical User Interface, the JavaTest Harness User's Guide: Command-Line Interface, or the JavaTest harness online help for a detailed description specifying keywords used when running tests. See A.1.3 Keywords for more information about keywords.

4.4.5 Configuration Question Log

When the Configuration Editor saves a configuration, it generates a Configuration Question Log that lists each question asked, the tag name of the question, and the corresponding answer.

The Configuration Question Log can be used during troubleshooting to quickly review the contents of the current configuration interview.

See the JavaTest Harness User's Guide: Graphical User Interface or the JavaTest harness online help for a detailed description of using the Configuration Question Log.

4.4.6 Configuration Checklist

When the Configuration Editor saves a configuration, it generates a Configuration Checklist from each interview. Some tests require additional setup steps (such as starting additional programs) prior to running them. You can use the Configuration Checklist to verify that all required preparations are complete before running tests.


Note - Because the Configuration Editor generates the checklist from the completed interview, it only lists those setup steps actually required by that configuration to run tests.


See the JavaTest Harness User's Guide: Graphical User Interface or the JavaTest harness online help for a detailed description of using the Configuration Checklist.

4.5 Special Setup Instructions

Special setup instructions are described on a test-by-test basis in Chapter 5 as well as in the Configuration Checklist. See 4.4.6 Configuration Checklist for information about the Configuration Checklist. Table 4-23 lists the tests that require additional setup and the page containing the setup instructions.

Table 4-23 Special Test Setup Instructions

Test

Section and Page Number

Annotation Processing Tests

5.2.1 Setup

AWT and Swing Tests

5.3.1 Setup

Compiler Tests

5.4.1 Setup

CORBA Tests

5.5.1 Setup

Distributed Tests

5.6.1 Setup

Extra-Attribute Tests

5.7.1 Setup

ImageIO Tests

5.9.1 Setup

Interactive Tests

5.10.1 Setup

JAX-WS Mapping Tests

5.11.1 Setup

JDBC Technology RowSet Tests

5.12.1 Setup

JDWP Tests

5.13.1 Setup

JMX Advanced Dynamic Loading Tests

5.14.1 Setup

JMX Remote API Tests

5.15.1 Setup

JNI Implementation Tests

5.16.1 Setup

JVM TI Tests

5.17.1 Setup

Java Generic Security Service API Tests

5.19.1 Setup

Java Programming Language Instrumentation Services Tests

5.20.1 Setup

Java Platform Scripting API Tests

5.21.1 Setup

Java RMI Tests

5.23.1 Setup

Network Tests

5.25.1 Setup

Printing Tests

5.28.1 Setup

Schema Compiler

5.29.1 Setup

Schema Generator

5.30.1 Setup

Security Tests

5.31.1 Setup

Static Initialization Tests

5.33.1 Setup

4.6 Special Test Execution Requirements

The following tests require multiple runs using different configurations:

VM Tests — Test specific information required to run the VM tests in all required configurations is provided in the VM tests Execution section of Chapter 5.

4.7 JCK-devtools Test Suite Execution Requirements

This section includes special considerations, necessary steps, and preliminary tasks required to run JCK-devtools test suite tests.

  • Agent classpath should contain the path to the jtjck.jar file to run Schema Compiler, Schema Generator, and JAX-WS tests.

  • When you run schema compilation through the JavaTest Agent, your schema compiler will be executed in the Agent's JVM. Your implementation should provide a custom class, which implements the following interface:

    package com.sun.jck.lib;

    public interface SchemaCompilerTool {

    /**

    * @param xsdFiles array of strings containing schema files

    * @param packageName the target package

    * @param outDir output directory where java file(s) will be generated

    * @param out output stream for logging

    * @param err error stream for logging

    * @return 0 if java file(s) generated successfully

    */

    int compile(String[] xsdFiles, String packageName, File outDir, PrintStream out, PrintStream err);

    }

    Schema compilation should be accomplished during invocation of compile method. The JCK devtools test suite includes the following class that is fully compatible with Sun's reference implementation and that can be used for schema compilation: com.sun.jck.lib.SchemaCompiler. The realization of the method compile(String[] xsdFiles, String packageName, File outDir, PrintStream out, PrintStream err) compiles one or more schemas into java sources.

  • When you run schema generation through JavaTest Agent, your generator will be executed in the Agent's JVM. Your implementation should provide a custom class, which implements the following interface:

    package com.sun.jck.lib;

    public interface SchemaGenTool {

    int generate(String[] args, PrintStream out, PrintStream err);

    }

    The invocation of the generate method should start schema generation.

    The JCK devtools test suite includes the following class that is fully compatible with Sun's reference implementation and that can be used for schema generation: com.sun.jck.lib.SchemaGen. The realization of the method generate(String[] args, java.io.PrintStream out, java.io.PrintStream err) generates xml schema from one or more java files.

  • JCK provides commands to invoke your schema compiler. For example, for the Sun JAXB implementation on Solaris, Sun provides a script solaris/bin/xjc.sh that compiles schemas into class files, the script can be found at the following location:

    solaris/bin/xjc.sh

    For the Sun JAXB implementation on Windows, Sun provides a script win32\bin\xjc.bat, which compiles schemas into class files, the command can be found at the following location:

    win32\bin\xjc.bat

    The xjc script supports the following mandatory command-line options:

    Usage: xjc –cp <arg> –p <pkg> –d <dir> <schema file>

    Options:

    • –cp <arg> — specifies class search path

    • –p <pkg> — specifies the target package

    • –d <dir> — target directory to store generated files

  • JCK provides commands to invoke your schema generator. For example, for the Sun JAXB implementation on Solaris, Sun provides a script solaris/bin/schemagen.sh that generates schemas from java files, the script can be found at the following location:

    solaris/bin/schemagen.sh

    For the Sun JAXB implementation on Windows, Sun provides a script win32\bin\schemagen.bat that compiles schemas into class files, the script can be found at the following location:

    win32\bin\schemagen.bat

    The schemagen script supports the following mandatory command-line options:

    Usage: schemagen –cp <arg> –d <dir> <java files>

    Options:

    • –cp <arg> — specifies class search path

    • –d <dir> — target directory to store generated files

Previous Contents Index Next
Company Info Contact Terms of Use Privacy Copyright 1994-2008 Sun Microsystems, Inc.