JavaTest Harness Tutorial
This tutorial introduces the JavaTest harness GUI and some basic
underlying concepts. The tutorial instructs you to run a small test
suite called demotck. The demotck test suite
contains 17 tests that test the functionality of some very simple
This tutorial describes how to do the following steps:
Use version 5.0 (or later) of the Sun Java SE platform on either
the Microsoft Windows (Win32) or Solaris Operating System.
Note - Unless otherwise indicated, all examples in this
book use Microsoft Windows-style command prompts and file
D.1 Starting the JavaTest Harness
To keep things simple, these instructions show you how to run
both the JavaTest harness and the tests on the same system in
different VMs (see Figure D-1).
Figure D-1 JavaTest Harness and Tests Running on Same
It is also possible to run the JavaTest harness and the tests on
separate systems using the JavaTest harness agent.
To Start the
- Verify that Java SE 6 software is in your path.
At a command prompt, enter:
C:\> java -version
the current directory.
Where jck is the JCK installation directory.
Note - Run the JavaTest harness on a computer with a
minimum of 256 megabytes of physical memory. Use an agent with the
JavaTest harness to run test programs on platforms with limited
amounts of memory. If you are running a large test suite (10,000 or
more tests), you must allocate a large amount of memory to the VM
to run the JavaTest harness properly. Allocate 512 megabytes when
running a large test suite by adding -Xmx512m to the
command string when you start the JavaTest harness from a writable
directory. When allocating memory to the VM, do not exceed the
actual amount of memory available on your system.
- Start the JavaTest harness.
On UNIX systems, enter the following command at the command
java -Xmx512m -jar lib/javatest.jar -newDesktop
On Win32 systems, enter the following command at the command
java -Xmx512m -jar lib\javatest.jar -newDesktop
Note - The -newDesktop option is used here to
ensure that the JavaTest harness starts exactly as described in
these instructions. Under normal circumstances do not use
this option. When you use this option, you lose any information
that the harness saved about your previous session. For information
about JavaTest harness options, see the JavaTest harness online
The JavaTest harness starts and displays the Quick Start wizard
window as shown in Figure D-2.
Figure D-2 The JavaTest Harness With Quick Start
D.2 Using the Quick Start Wizard
The Quick Start wizard leads you through the basic steps
required to start running the test suite.
To Use the Quick
- Choose Start a new test run on the Welcome to the JavaTest
Harness screen. Click Next
- Choose Create a new configuration on the Configuration
screen. Click Next.
- Click Next on the Test Suite screen to accept the default
test suite setting.
- Click Browse on the Work Directory screen. Click Next.
The JavaTest harness uses the work directory to store
information and to write test results. Use the file chooser to
create the work directory in a convenient location outside of
the test suite directory (demotck).
- Click Finish on the Almost Done screen to accept the default
setting and complete the Quick Start process.
The default setting automatically starts the Configuration
Editor when the Quick Start wizard closes.
D.3 Configuring Test Information
Because you used the default setting in the last panel of the
Quick Start wizard, the Configuration Editor starts
Use the Configuration Editor to configure the information
required to run the test suite. The Configuration Editor consists
of three panes and a menu bar, as shown in Figure D-3.
Figure D-3 JavaTest Harness Configuration Editor
The left pane lists the titles of the questions you have
answered, are currently answering, or that the editor deduces must
be answered. The current question is highlighted.
The center pane displays the interview questions. Answer the
questions by using controls such as text boxes, radio buttons, or
drop-down menus located below each question. Whenever possible, the
editor deduces answers from your system configuration and includes
them in text boxes, combo boxes, and radio buttons. You can accept
these answers or provide other answers.
The right pane displays important information about each
question, including background information, examples of answers,
and additional information about choosing an answer.
Answer the questions displayed in the configuration editor.
The Demo interview contains 13
questions. Table D-1 presents the titles,
answers, and information about each question that you must answer
in the interview.
Table D-1 Tutorial Interview Questions and Answers
Briefly describes the purpose and function of the Demo
Names the interview file.
Describes the configuration.
How to Run Tests
On this computer
Runs both the JavaTest harness and the tests on the same
Java Virtual Machine
The absolute path to the java command on a win32
Click Browse to activate a file chooser, or type the path
directly in the text box.
Causes all executing tests to emit standard information
Introduces the section of questions about the tests to run and
how to run them.
Specify Tests to Run?
Runs all of the tests.
Specify an Exclude List?
Specifies that an exclude list is not used for this test
Specifies that prior run status is not used to filter the test
run. Feel free to try it on subsequent runs.
Specifies the default concurrency setting (1).
Specifies the default standard time out value for each test
The configuration editor has all of the information it needs to
run the tests.
Click the Done button to save the interview. JavaTest harness
interviews are saved to files that end with the .jti
suffix. Use the file chooser to specify a file in a convenient
D.4 Running Tests
To Run the
- Change the View Filter setting in the tool bar from All
Tests to Last Test Run.
Choose Last Test Run in the View Filter drop-down menu located
in the tool bar. See Figure D-4, for the
location of the View Filter combo box. This changes your view of
the test tree so that you only see the results of the current test
run. This is generally the view that most users prefer. Refer to
the JavaTest Harness User's Guide: Graphical User Interface
for a detailed description of the JavaTest harness view
Note - When you change to the Last Run filter before you
do a test run, the folders and tests in the tree turn to gray,
indicating that they are filtered out. This occurs because results
fo not exist from your last test run.
- Choose Run Tests -> Start to start the test run.
The harness begins to run tests. You can see activity in the
test tree panel that indicates the tests currently running. You can
also watch the progress of the test run in the progress monitor on
the bottom-right portion of the JavaTest harness window and the pie
chart in the Summary tab.
- Expand the test tree folders to reveal the tests.
Click different test folders to expand the test tree. See
Figure D-4 for an example of an expanded
Figure D-4 Expanded Test Tree Example
D.4.1 JavaTest Harness GUI — Folder and Test Colors
As tests complete, the tests and their folders change color to
represent their state.
Table D-2 briefly describes the colors
and their meaning.
Table D-2 Folder and Test Colors and Their Meaning
Error. The test could not be run properly. Usually indicates a
Filtered out. Due to a parameter setting (for example, it is on
an exclude list), the test is not selected to be run.
Folders reflect the state of the tests hierarchically beneath
them. You know that the entire test suite passed if the test suite
root folder is green. See the JavaTest harness online help for more
Note - The test
contains errors and is supposed to fail as part of the tutorial. If
any other tests fail, check your answers to the configuration
D.5 Browsing Test Results
When the test run is complete, use the Folder tabbed pane and
Test tabbed pane portion of the JavaTest harness to examine the
results. Examine the output of the test that failed.
Note - The Folder tabbed pane and the Test tabbed pane
occupy the same portion of the Test Manager window. The Folder
tabbed pane is displayed when you choose a folder entry in the test
tree and the Test tabbed pane is displayed when you choose a test
entry in the test tree.
D.5.1 The Folder Pane
The Folder tabbed pane displays information about the tests in
the selected folder. Figure D-5 shows an
example of the folder pane.
Figure D-5 Folder Pane Example
Using the Folder
- Click the top folder in the test tree (the test tree
- Click the Summary tab (shown by default) to display the
Notice the statistics displayed in the Summary panel. It
describes how many tests in the test suite passed, failed, had
errors, and were filtered out.
- Click any of the other folder icons in the test tree to
display the statistics for the selected tests.
This action causes the Summary panel to display the statistics
for all tests hierarchically beneath the selected folder.
- Click the test tree root folder again to redisplay the test
This action causes the Summary panel to redisplay the statistics
for the complete test suite.
- Click the Passed tab to display a list of tests that passed
during the test run.
- Click the Failed tab to display a list of the tests that
failed during the test run (only one test in this case).
- Double-click the
lists\DoublyLinkedList\InsertTest.html test in the Failed
tab to change the display from the Folder pane to the Test
This action selects the test in the test tree and changes the
display from the Folder pane to the Test pane.
Note - To read more information about any of the panes,
click on a tab to establish focus and press F1 to activate online
help about that pane.
D.5.2 The Test Pane
The Test tabbed pane displays information about the selected
test. The five tabs provide information about the test and
information about the results of its execution. Figure D-6 shows an example of the test pane.
Figure D-6 Test Pane Example
Each tab displays information collected by the JavaTest harness
about the test highlighted in the test tree.
Table D-3 briefly describes each
Table D-3 Test Pane Tabs
Test Run Messages
Displays messages generated during the selected test's
Test Run Details
A table of values generated during the selected test's
A table of the configuration values used during the selected
Displays the source code and any other files related to the
A table of the test description values specified for the
Note - To read more information about any of the panes,
click on a tab to establish focus, and press F1 to activate the
online help about that pane.
Using the Test
- Click the Test Run Messages tab to view the messages
generated by the JavaTest harness or the test during execution.
Notice that the red icons indicate that the test failed.
- Click the execute messages item in the left column to
display the command line used to run the test.
The display on the right shows the command line used to run the
test. Problems can often be debugged by examining how the test was
invoked. In this case it was invoked correctly. See Figure D-7 for an example of the test messages
Figure D-7 Test Messages Pane Example
- Click the out1 item in the left column to display errors
reported by the test.
The display on the right shows errors reported by the test. The
messages indicate that either the test or the API contain errors.
In this example, the test contains errors. Figure D-8 shows an example of logged error
messages in the test messages pane.
Figure D-8 Logged Error Messages Example
D.6 Excluding a Failed Test
The JavaTest harness allows you to exclude tests from a test
suite by specifying an exclude list file. This section shows you how to use the
Quick Set Mode of the Configuration Editor to specify an exclude
list that includes lists\DoublyLinkedList\InsertTest.html.
Tests that are excluded are not executed during test runs, and
though they are still displayed in the test tree, their status is
not reflected in the pass-fail status of the test suite.
To Exclude a Failed
- Choose Configure -> Change Configuration -> Exclude
List from the Test Manager menu bar.
The Configuration Editor opens directly to a panel that enables
you to specify an exclude list. Quick Set Mode allows you to
quickly change values that change frequently between test runs.
These values are also referred to as standard values.
Standard values can also be changed using the Configuration Editor
in Question Mode.
- In the Exclude List pane, click Other.
This activates a tool used to specify a set of exclude
- Click the Add button on the upper-right portion of the
This invokes a file chooser used to specify an exclude list. The
current directory of the file chooser is the directory that you
were in when starting the JavaTest harness. If it is not, navigate
to that directory.
- Double-click the lib directory entry in the file
- Choose the demo.jtx
entry in the file chooser and click Select.
Notice that the exclude list (demo.jtx) is added to the
Exclude Lists text box.
- Click Done in the Configuration Editor.
- Change the View Filter setting in the tool bar from Last
Test Run to Current Configuration.
Choose Current Configuration in the View Filter located in the
tool bar. Figure D-4 displays the
location of the View Filter. This changes your view of the test
tree so that it displays only the results of the tests selected and
filtered out in the configuration. This filter shows only the tests
that will run next, as opposed to the Last Test Run filter that
shows only the tests that have been run.
Notice that the icon for the InsertTest.html entry in
the Test tree changes from red to gray. This indicates that the
test is filtered out and will not be executed. Also notice that the
Test Suite Root folder changes from red to green, indicating that
all the currently selected tests passed.
D.7 Generating a Report
You can use the JavaTest harness to generate an HTML report that
describes the results of the test run. All of the information
contained in the report is available from the GUI. The following
steps describe how to generate and browse a report of the test run
completed in the previous sections of this tutorial.
To Generate a
- Choose Report -> Create Report.
The Create a New Report dialog box opens.
- Specify the directory where you want the JavaTest harness to
write the report files.
Use a file chooser to specify the directory and click on the
- Click the Create Report(s) button.
The reports are generated and you are asked whether you want to
view the report.
- Click Yes.
The reports are displayed in the JavaTest harness report browser
window. Scroll through the report and follow the various links to
view data about the test run.
- To print the report, open the report in your web browser and