This document is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 Unported License
  Previous Contents Index Next
Appendix A

JCK Test Specification

This appendix describes the structure of JCK tests and how they are executed. The appendix is divided into the following two sections:

  • A.1 Test Description — Contains the information that describes a JCK test.

  • A.2 Test Selection and Execution — Describes how tests are selected for a test run and the steps involved in executing compiler and runtime tests using the information in the test description.

A.1 Test Description

Every test in the JCK test suite has a corresponding test description. The test description is an HTML file or files that contain a test description table and test comments. Test descriptions can be organized in the following ways:

  • Test description table and comments for a single test contained in a single file

  • Test description tables and comments for a number of similar tests grouped together in a single file

  • Test description tables and comments in separate files linked together using hypertext links

Test descriptions can be viewed using a web browser.

A.1.1 Test Comments

Because it is formatted using HTML, the test description is easily read by computer programs, test developers, and users. All test descriptions in the JCK test suite contain comments that describe what the test does and how it works.

A.1.2 Test Description Table

The test description file contains the test description table. The test description table is an HTML table. Each row of the table contains a test description field. The values specified in the test description fields are used by the JavaTest harness when the tests are run.

See Table A-1 for an example of a test description table.

Table A-1 Sample Test Description Table

Item

Value

title

Checkbox - Checkbox Tests

name

Checkbox

source

CheckboxTests.java

executeClass

javasoft.sqe.tests.api.java.awt.Checkbox.CheckboxTests

keywords

runtime positive

executeArgs

-TestCaseID ALL

Table A-2 describes the test description fields used by JCK 6b:

Table A-2 Test Description Fields

Field

Description

title

A descriptive string that identifies what the test does. The title appears in reports and in the JavaTest harness status window.

name

A string that identifies the test.

source

For compiler tests, the source field contains the names of the files that are compiled during the test run.

For runtime tests, the source field contains the names of the files previously compiled to create the test's class files. Precompiled class files are included with the JCK. Source files are included for reference only. Source files are often .java files, but can also be .jasm or .jcod files.

  • .jasm is a low level bytecode assembler designed to assemble class files containing sets of bytecodes that are unusual or invalid for use in runtime tests

  • .jcod is a class level assembler designed to build classes with unusual or invalid structure for use in runtime tests

These tools are used to generate class files that cannot be reliably generated by a Java compiler.

For most XML parser tests in the JCK runtime, the source field contains the names of the files that are processed during the test run. These files are XML Schema sources and XML documents usually having file name extensions of .xsd and .xml respectively. Such tests share a single precompiled class, TestRun, that invokes the XML parser under test through the Java technology API and passes the source file names to the parser for processing.

The test model is similar to compiler testing because the sources used in the tests contain valid and invalid use of various constructs of corresponding languages.

keywords

String tokens that can be associated with a given test. They describe attributes or characteristics of the test (for example, how the test should be executed, and whether it is a positive or negative test). Keywords are often used to select or deselect tests from a test run.

Keywords are described in detail in A.1.3 Keywords.

executeClass

The name of the main tests's class that the JavaTest harness loads and runs. This class might in turn load other classes when the test is run.

executeNative

The name of the platform-native program used to execute this test. This is specific to a few JNI implementation tests that launch a VM from native C code.

executeArgs

An array of strings that are passed to the test classes being executed. The arguments might be fixed but often involve symbolic values that are substituted from the test environment (variables defined elsewhere in the test environment). The result of substituting values can be seen in the resulting .jtr files.

These arguments form the basis for the set of arguments that are passed into the tests defined in executeClass and executeNative fields.

The default value of any variable not defined in the test environment is an empty string.

rmicClasses

For Java RMI tests, this nominates the set of classes that are passed to the Java RMI compiler.

timeout

A value specified in seconds used to override the default ten minute timeout used with all JCK tests.

context

Specifies configuration values required by the test. The JavaTest harness checks to be sure these values are set before it runs the test and passes the values through to the test. If any of these values are not defined, the JavaTest harness does not run the test.

Context-sensitive properties are described in detail in A.1.4 Context-Sensitive Properties.

selectIf

Specifies a condition that must be satisfied for the test to be executed. This field is constructed using environment values, Java programming language literals, and the full set of Boolean operators:

(+, -, *, /, <, >, <=, >=, &, |, !, !=, ==).

Example:

integerValue>=4 & display=="my_computer:0"

If the Boolean expression evaluates to false, the test is not run. If the expression evaluates to true, the test is run. If any of values are not defined in the test environment, the JavaTest harness considers the test to be in error.

remote

This entry contains information required to run distributed network tests.

jplisAgent

Specifies one Java PLIS agent. Following is the field value format:

agent-class-name options

  • agent-clas-name is the fully-qualified class name of the JPLIS agent.

  • options is a nonblank-space separated string that represents options to be passed to the agent specified by agent-class-name.

One test description table might be several jplisAgent fields. The jplisAgent field is processed only if the executed test has the jplis keyword in the list of keywords in the test description table.

apClass

The name of the annotation processor tests' class that is passed to the tested Java compiler.

apSource

Contains the names of files previously compiled to create the annotation processor's class files. The precompiled class files are included with the JCK.

apArgs

The annotation processor specific option value that is passed to the tested Java compiler. The value of this field is stored in the jckApArgs option name. The following is an example of passing the option with the standard JDK on Solaris:

-AjckApArgs=$apArgs.

A.1.3 Keywords

Keywords are tokens associated with specific tests. Keywords have two functions:

  • Convey information to the JavaTest harness about how to execute the tests.

  • Serve as a basis for including and excluding tests during test runs. You can specify keyword expressions in the JavaTest harness Configuration Editor keywords field to filter tests during test runs (see Chapter 4).

Keywords are specified in the keywords field of the test description. The set of valid keywords for JCK 6b is described in Table A-3 and Table A-4.

Table A-3 describes JCK keywords used in a wide variety of tests.

Table A-3 General Keywords

Field

Description

compiler

Identifies tests used with compiler products.

interactive

Identifies tests that require human interaction.

negative

The component under test must terminate with (and detect) an error. An operation performed by a negative test on the component under test must not succeed.

only_once

Identifies tests that can only be run one time in the same VM. While tests are usually run once, it is possible that tests could be run more often. Tests with this keyword fail to run correctly if run more than once in the same VM.

positive

The component under test must terminate normally. An operation performed by the test on the component under test must succeed.

robot

If your system supports the awt.robot API, the tests can be run automatically. Otherwise, they run interactively.

runtime

Identifies tests used with runtime products.

serial

Used only with compiler tests. Identifies a test whose source files must be compiled one at a time by separate compiler invocations.

If serial is not specified, the source files are compiled by group (.java).

Table A-4 describes keywords used only in specific tests.

Table A-4 Test Specific Keywords

Field

Description

distributed

Identifies distributed tests that validate functionality while the test platform is connected to a remote host.

idl_inherit

Identifies tests that require the IDL-to-Java programming language translator to output Java technology interfaces that use the inheritance object model.

idl_tie

Identifies tests that require the IDL-to-Java programming language translator to output Java technology interfaces that use the TIE object model.

java_to_schema

Identifies JAXB tests that validate Java-to-Schema mapping.

jaxb

Identifies JAXB tests that validate API and Java-to-Schema mapping.

jaxws

Identifies tests that validate JAX-WS functionality.

jniinvocationapi

Identifies tests that use the JNI implementation.

jplis

Identifies JPLIS tests that require separate execution in single-VM mode and special options in the tested Java technology command line.

jplislivephase

Identifies JPLIS Live Phase tests that require separate execution in single-VM mode and that require support of an optional mechanism to start instrumentation agents some time after the VM has started without specifying agents through Java platform command line options.

jvmti

Identifies JVM TI tests that require separate execution in single-VM mode and special options in the tested Java technology command line.

jvmtilivephase

Identifies JVM TI Live Phase tests that require separate execution in single-VM mode and that require support of an optional mechanism to start instrumentation agents some time after the VM has started without specifying agents through Java platform command line options.

jdwp

Identifies JDWP tests that require separate execution in single-VM mode and special options in the tested Java technology command line.

needsharedclassloader

Identifies tests that must be loaded by the same class loader when they are run in single-VM mode. Tests with this keyword must use the same native library due to the Java SE limitation that does not allow the same native library to be loaded by different class loaders.

rmi_v11

Identifies RMI compiler tests that require version 1.1 stubs or skeletons to be produced by the RMI compiler.

rmi_iiop

Identifies RMI tests that use the IIOP transport layer.

A.1.4 Context-Sensitive Properties

A number of tests require information about the particular Java platform technology under test (context) to determine expected results. This information is identified by entry names in the context field of the test description table. The JavaTest harness checks to be sure that all values specified in the context field are defined in the test environment before it attempts to execute the test. If any of the values are not set, the test is not executed and the test is considered to be in error.

The valid context-sensitive properties for JCK 6b are described in the following sections.

A.1.4.1 Network Resources

Table A-5 lists network resources, which are properties that the network tests use, such as the IP addresses of systems used during the tests.

Table A-5 Network Resources

Field

Data Format

Description

network.fileURL

URL

A file URL for the network tests.

For example:

file:///C:\\AUTOEXEC.BAT

network.ftpURL

URL

An FTP URL for the network tests.

For example:

ftp://javator/etc/hosts

network.httpURL

URL

An HTTP URL for the network tests to use.

For example:

http://www.sun.com

network.jckHTTPProxyHost

Proxy name

The name of the HTTP proxy server to be used by JCK tests if a direct connection is not used.

For example:

fproxy.somewhere.com

network.jckHTTPProxyPort

Port number

The port number of the HTTP proxy server to be used by JCK tests if a direct connection is not used.

For example:

8080

network.localHost

Hostname and IP address

The name and IP address of the local host.

For example:

ruffles/129.144.75.50

network.tcpPortRange

Range of ports

On some systems, the set of ports allocated by the OS is different from the set of ports that can be specifically requested by client code. In this case, network.tcpPortRange must be set to indicate to the test the range of ports that are available. The lower limit is 1024 and the upper limit is 65535.network.tcpPortRange test configuration value

For example:

2048-5096

network.testHost1

Hostname and IP address

A host used for network tests.

For example:

satchmo/129.144.250.133

network.testHost2

Hostname and IP address

Another test host (different from testHost1).

For example:

doppio/129.144.173.35

network.udpPortRange

Range of ports

On some systems, the set of ports that are allocated by the OS is different from the set of ports that can be specifically requested by client code. In this case, network.udpPortRange must be set to indicate to the test the range of ports that are available. The lower limit is 1024 and the upper limit is 65535.

For example:

2048-5096

A.1.4.2 Remote Resources

Many network tests communicate with an additional system (distributed test server) during the test run. These tests are referred to as distributed tests. Table A-6 identifies and describes the configuration values that must be set for the distributed tests to work.

Table A-6 Remote Resources

Name

Data Format

Description

remote.networkAgent

-host hostname

-port port number (optional)

-classpath classpath (optional)

The distributed server to use with distributed tests.

For example:

-host doppio

-port 345

-classpath D:\classess

A.1.4.3 Hardware Characteristics

Table A-7 identifies and describes the features that differ based on hardware characteristics.

Table A-7 Hardware Characteristics

Name

Format

Description

hardware.xFP_ExponentRanges

Integer

Runtime argument for floating-point tests. For more information, see Chapter 5.

For example:

-16382:16383:-16382:163831

1Note: These values are only valid for Java technology runtime implementations that use only the standard formats for floating-point operations.

A.1.4.4 Platform Characteristics

Table A-8 identifies and describes the features that might or might not be present on a particular hardware and software platform.

Table A-8 Platform Characteristics

Name

Format

Description

platform.canPlayMidi

Boolean

Specifies if the system can play MIDI encoded sound.

For example:

false

platform.canPlaySound

Boolean

Specifies if the system supports the ability to play sampled audio.

For example:

true

platform.canRecordSound

Boolean

Specifies if the system supports the ability to record sound.

For example:

true

platform.expectOutOfMemory

Boolean

Indicates if the tested system can throw an OutOfMemoryError.

For example:

true

platform.hasPrinter

Boolean

Indicates if a printer is connected to the tested system. If a printer is connected to the system and is online, the value must be set to true.

For example:

true

platform.isFallbackSupported

Boolean

Indicates if the system performs the verification by the type inference when the typechecking fails.

For example:

true

platform.isHeadless

Boolean

Indicates if the tested system is headless (does not have a display device, a keyboard, or mouse).

For example:

false

platform.JavaSecurityAuthPolicy

String

Specifies the mechanism your product uses to specify the Java platform Security Authorization Policy used to run JAAS tests.

For example:

java.security.auth.policy

platform.JavaSecurityAuthLoginConfig

String

Specifies the mechanism your product uses to specify the Java platform Security Login Configuration used to run JAAS tests.

For example:

java.security.auth.login.config

platform.JavaSecurityKrb5KDC

String

VM System Property to set the Java platform Security Kerberos Key Distribution Center.

For example:

java.security.krb5.kdc

platform.JavaSecurityKrb5Realm

String

VM System Property to set the Java platform Security Kerberos Key Distribution Center realm.

For example:

java.security.krb5.realm

platform.KDCHostName

String

Host name of the Kerberos Key Distribution Center.

For example:

plop.sfbay.sun.com

platform.jdwpSupported

Boolean

Indicates if the JDWP implementation is provided on your system.

For example:

true

platform.jvmtiSupported

Boolean

Indicates if the system supports JVM TI.

For example:

true

platform.MultiVM

Boolean

Specifies if your system can load and run separate instances of a tested VM.

For example:

false

platform.KDCRealm

String

Kerberos realm name containing the following two accounts that are operating.

For example:

PLOP

platform.KerberosClientPassword

String

Password for the second principal.

For example:

xxxxxxx

platform.KerberosClientUsername

String

Second Kerberos principal name.

For example:

client

platform.KerberosServerUsername

String

First Kerberos principal name.

For example:

server

platform.KerberosServerPassword

String

Password for the first principal.

For example:

xxxxxxx

platform.Krb5LoginModuleConfigured

Boolean

Indicator whether module com.sun.security.auth.module. Krb5LoginModule is configured in the JRE software.

For example:

true

platform.maxMemory

Positive double or zero with optional postfix m or k

Amount of memory a test might allocate (in megabytes, kilobytes, or bytes).

For example:

205m

platform.nativeCodeSupported

Boolean

Indicates if the system supports direct access to native code (including but not limited to JNI implementation).

For example:

true

platform.robotAvailable

Boolean

Indicates if the system supports low-level input control (provided by java.awt.robot classes)

For example:

true

platform.rowSetFactory

String

The fully qualified name of the factory class used for creating the rowsets.

latform.soundURL

URL

The URL path to a sound file available to the system. The URL is used by various sound tests for playback and conversion testing. The URL should be a sound file (wav, au, or snd) that your system is capable of playing.

Example:

file:

$testSuiteRootDir$/api$/javax_sound $/sound.wav

platform.staticSigTestClasspath

Classpath

List of directories and ZIP or JAR files where the classes tested by the Static Signature Test are located, separated with a system-dependent path separator.

Example:

/usr/local/java/solsparc/jre /lib/rt.jar

platform.supportStaticSigTest

Boolean

Boolean value that indicates whether the static signature test is executed on the system being tested. The static signature test must be executed if the class files for the JRE software are delivered in a directory hierarchy, or in a ZIP or JAR file.

Example:

true

A.2 Test Selection and Execution

This section describes how JCK tests are selected for a test run and how they are then executed.

A.2.1 Test Results

Table A-9 describes the three states used to report test execution results.

Table A-9 Test Execution Result States

State

Description

Pass

A test passes when the functionality being tested behaves as expected. All tests are expected to have passing results.

Fail

A test has failing results when the functionality being tested does not behave as expected.

Error

A test is considered to be in error when something (usually a configuration problem) keeps the test from being executed as expected. Errors often indicate a systemic problem. A single configuration problem can cause many tests to fail. For example, if the path to the runtime is configured incorrectly, the harness cannot run tests and identifies them as having errors.

A.2.2 Tests Selected for a Test Run

Table A-10 describes the factors used by the JavaTest harness to select tests immediately prior to starting a test run.

Table A-10 Test Selection Factors

Selection Factor

Description

Tests (JavaTest harness GUI)

The JavaTest harness finds tests listed in the Tests to be Run field of the Configuration Editor. You can specify sub-branches of the tree as a way of limiting the tests executed during a test run. The JavaTest harness walks the tree starting with the sub-branches or tests you specify and executes all tests that it finds. See Chapter 4 for more information.

Exclude list (JavaTest harness GUI)

Tests listed in the appropriate exclude list are deselected prior to the start of a test run. For details about exclude lists and their role in the certification process, see Chapter 2. For a description about how to specify an exclude list using the Configuration Editor, see Chapter 4.

Keywords (JavaTest harness GUI)

A test can be selected based on keywords specified in the keywords field in the test description. JCK keywords are described in detail in A.1.3 Keywords. For a description about how to use keywords with the JavaTest harness Configuration Editor, see Chapter 4.

Prior Status

Use the checkboxes to select tests in a test run based on their outcome on a prior test run. Prior status is evaluated on a test-by-test basis using information stored in result files (.jtr) written in the work directory.

selectIf field (Test Description table)

The selectIf field in the test description table contains an expression composed of elements from the test environment, Java programming language literals, and the standard operators:

+, -, *, /, <, >, <=, >=, &, |, !, !=, ==

For example:

integerValue>=4 & display=="my_computer:0"

These expressions are evaluated prior to the start of the test run. If the expression evaluates to true, then the test is selected as part of the test run. If the expression evaluates to false, or if any of the elements are not defined in the test environment, the test is not selected.

A.2.3 Compiler Test Execution

The section describes how JCK compiler tests are executed. Compiler tests contain the compiler keyword in the test description. They test either the Java compileror the RMI compiler.

A.2.3.1 Java Compiler Tests

All Java compiler test source files are .java files.

If the apClass field is specified in the test description, the annotation processor class name in this field is passed to the Java compiler and to those test source files not specified in the apSource test description field.

If the test description keywords field contains the serial keyword, the source files are compiled one at a time in the order they are listed in the test description source field. If the serial keyword is not specified, they are compiled as a group.

If the negative keyword is specified in the test description, the test acquires the following result status:

  • Fails if the compilation succeeds

  • Passes if the compilation fails

If the positive keyword is specified in the test description, the test acquires the following result status:

  • Fails if the compilation does not succeed

  • If the compilation succeeds, the resulting classes are executed on a reference runtime, and the test acquires the following result status:

    • Fails if execution does not succeed

    • Passes if execution succeeds


Note - Compilation is carried out using the command.testCompile.java entry from the test environment. The variable $testSources is set to the source files listed in the test description source field.



Note - Classes are executed using the command command.refExecute. The $testExecuteClass variable is set to the class to execute, the $testExecuteArgsvariable is set to the value of the arguments to be passed to the class to execute. See A.2.4.1 testExecuteArgs Variable for more information.


Figure A-1 illustrates the execution model.

Figure A-1 Java Class File Compilation Test Flow Diagram
Java Class File Compilation Test Flow Diagram
A.2.3.2 RMI Compiler Tests

Two steps are performed for RMI compiler (RMIC) tests.

  1. Java programming language sources are compiled into Java class files.

    If the Java programming language compilation does not succeed, the RMIC step is not run and the JavaTest harness considers the test to be in error.


    Note - For notes about compiling Java class files, see A.2.3.1 Java Compiler Tests.


  2. Some or all of the resulting class files are then compiled by the RMIC.

    The test description field of the rmicClasses contains the list of classes to be compiled by the RMIC.

    If the negative keyword is specified in the test description, the test acquires the following result status:

    • Fails if the RMIC compilation succeeds

    • Passes if the RMIC compilation does not succeed

    If the positive keyword is specified in the test description and the compilation does not succeed, the test fails.

    If the positive keyword is specified in the test description and the compilation succeeds, the resulting class files are executed on a Reference Runtime. The test acquires the following result status:

    • Fails if execution does not succeed

    • Passes if execution succeeds

    The RMIC compilation is carried out using one of two commands.

    If the rmi-iiop keyword is specified, use the following command:

    command.testRMIC.iiop
    

    If the rmi-iiop keyword is not specified, use the following command:

    command.testRMIC
    

    The list of class files to be compiled is derived from the list of class files specified in the rmicClasses test description field and is contained in the $testRmicClasses variable.

    Figure A-2 illustrates the execution model.


    Note - Test class files are executed using the command command.refExecute. The $testExecuteClass variable is set to the class file to execute, the $testExecuteArgs variable is set to the value of the arguments to be passed to the class file to execute. See A.2.4.1 testExecuteArgs Variable for more information.


Figure A-2 RMI Compilation Test Flow Diagram
RMI Compilation Test Flow Diagram

A.2.4 Runtime Tests

Runtime tests are all tests that contain the runtime keyword in the test description.

Runtime tests might be written in the Java programming language or in one of the two byte code assemblers (.jasm and .jcod) used to construct class files with specific characteristics:

  • .jasm is a low-level bytecode assembler designed to assemble sets of bytecodes that are unusual or invalid for use in runtime tests.

  • .jcod is a class-level assembler designed to build invalid classes for use in runtime tests.

Runtime tests are packaged as compiled class files although the sources are also included for reference purposes.

Executing Runtime Tests
  1. If the test description table contains remote fields, the test is a distributed test, the following applies:
    1. A message switch is started to enable communication between the components running remotely.

      All communication between these distributed components goes through the host running the JavaTest harness. The value of $testMsgSwitch is set to the name of the machine running the JavaTest harness and a port on that machine to use for the communication:

      $testMsgSwitch=host:port
      
    2. Each of the remote entries is activated.

      Note - For more information about the remote field, see Table A-2.


  2. The main class file is executed (both distributed and non-distributed tests).

    If execution results in an error, the test is in error. An error indicates that the test code was run incorrectly and is an unintended result, as opposed to a failure that could be an intended result (see Step 4).

  3. If the test is a distributed test, the message switch is closed down, and the results from the remote components are collected and combined with the result of the main class file.

    The final status is the "first worst" status: failed is worse than pass, error is worse than failed).

  4. Positive and negative checks.

    If the test description contains the negative keyword the test acquires the following result status:

    • Passes if execution fails (Step 2). Few negative runtime tests are used. A few tests are required to test invalid main(...) signatures.

    • Fails if execution succeeds (Step 2).

    If the test has a positive keyword (the common case) the result of the test is the same as the result of the execution (Step 2 for local tests, Step 3 for distributed tests).


    Note - Test class files are executed using the command command.testExecute. The $testExecuteClass variable is set to the class file to execute, the $testExecuteArgs variable is set to the value of the arguments to be passed to the class file to execute. See A.2.4.1 testExecuteArgs Variable for more information.


    Figure A-3 illustrates the execution model.

    Figure A-3 Runtime Test Flow Diagram
    Runtime Test Flow Diagram
A.2.4.1 testExecuteArgs Variable

Whenever the JavaTest harness executes a test class file it obtains arguments required by the class file (if any exist) from the $testExecuteArgs variable in the test environment. The $testExecuteArgs variable is constructed from a variety of sources and assembled in the following order:

  1. Some runtime tests include a number of test cases. Some of these tests might be excluded. If the exclude list identifies cases to be excluded, these cases are passed into the test using the -exclude option.

    For example:

    -exclude testcase1,testcase2,...
    
  2. For each entry EEE in the context field of the test description table, the JavaTest harness looks up EEE in the environment. If the name is not found, the test is automatically in error, otherwise $testExecuteArgs has an entry equivalent to the following:

    -EEE $EEE
    
  3. Any values supplied to the executeArgs field of the test description table are added to $testExecuteArgs. These values have any $... entries resolved in the environment, but no errors occur if any of these values are not defined.

Previous Contents Index Next
Company Info Contact Terms of Use Privacy Copyright 1994-2008 Sun Microsystems, Inc.