JavaTest Harness Tutorial

This tutorial introduces the JavaTest harness GUI and some basic underlying concepts. The tutorial instructs you to run a small test suite called demotck. The demotck test suite contains 17 tests that test the functionality of some very simple demo APIs.

This tutorial describes how to do the following steps:

• D.1 Starting the JavaTest Harness

• D.2 Using the Quick Start Wizard

• D.3 Configuring Test Information

• D.4 Running Tests

• D.5 Browsing Test Results

• D.6 Excluding a Failed Test

• D.7 Generating a Report

Use version 5.0 (or later) of the Sun Java SE platform on either the Microsoft Windows (Win32) or Solaris Operating System.

Note - Unless otherwise indicated, all examples in this book use Microsoft Windows-style command prompts and file separators.

D.1 Starting the JavaTest Harness

To keep things simple, these instructions show you how to run both the JavaTest harness and the tests on the same system in different VMs (see Figure D-1).

Figure D-1 JavaTest Harness and Tests Running on Same System

It is also possible to run the JavaTest harness and the tests on separate systems using the JavaTest harness agent.

To Start the JavaTest Harness

1. Verify that Java SE 6 software is in your path.

   At a command prompt, enter:

   C:\> java -version
   

2. Make <jck>\doc\javatest\tutorial\simpleHTML\demotck the current directory.

   Where jck is the JCK installation directory.

   ---

   Note - Run the JavaTest harness on a computer with a minimum of 256 megabytes of physical memory. Use an agent with the JavaTest harness to run test programs on platforms with limited amounts of memory. If you are running a large test suite (10,000 or more tests), you must allocate a large amount of memory to the VM to run the JavaTest harness properly. Allocate 512 megabytes when running a large test suite by adding -Xmx512m to the command string when you start the JavaTest harness from a writable directory. When allocating memory to the VM, do not exceed the actual amount of memory available on your system.

   ---

3. Start the JavaTest harness.

   On UNIX systems, enter the following command at the command prompt:

   java -Xmx512m -jar lib/javatest.jar -newDesktop
   

   On Win32 systems, enter the following command at the command prompt:

   java -Xmx512m -jar lib\javatest.jar -newDesktop
   

   ---

   Note - The -newDesktop option is used here to ensure that the JavaTest harness starts exactly as described in these instructions. Under normal circumstances do not use this option. When you use this option, you lose any information that the harness saved about your previous session. For information about JavaTest harness options, see the JavaTest harness online help.

   ---

   The JavaTest harness starts and displays the Quick Start wizard window as shown in Figure D-2.

   Figure D-2 The JavaTest Harness With Quick Start Wizard

   

D.2 Using the Quick Start Wizard

The Quick Start wizard leads you through the basic steps required to start running the test suite.

To Use the Quick Start Wizard

1. Choose Start a new test run on the Welcome to the JavaTest Harness screen. Click Next
2. Choose Create a new configuration on the Configuration screen. Click Next.
3. Click Next on the Test Suite screen to accept the default test suite setting.
4. Click Browse on the Work Directory screen. Click Next.

   The JavaTest harness uses the work directory to store information and to write test results. Use the file chooser to create the work directory in a convenient location outside of the test suite directory (demotck).

5. Click Finish on the Almost Done screen to accept the default setting and complete the Quick Start process.

   The default setting automatically starts the Configuration Editor when the Quick Start wizard closes.

D.3 Configuring Test Information

Because you used the default setting in the last panel of the Quick Start wizard, the Configuration Editor starts automatically.

Use the Configuration Editor to configure the information required to run the test suite. The Configuration Editor consists of three panes and a menu bar, as shown in Figure D-3.

Figure D-3 JavaTest Harness Configuration Editor Example

The left pane lists the titles of the questions you have answered, are currently answering, or that the editor deduces must be answered. The current question is highlighted.

The center pane displays the interview questions. Answer the questions by using controls such as text boxes, radio buttons, or drop-down menus located below each question. Whenever possible, the editor deduces answers from your system configuration and includes them in text boxes, combo boxes, and radio buttons. You can accept these answers or provide other answers.

The right pane displays important information about each question, including background information, examples of answers, and additional information about choosing an answer.

• Answer the questions displayed in the configuration editor.

  The Demo interview contains 13 questions. Table D-1 presents the titles, answers, and information about each question that you must answer in the interview.

Table D-1 Tutorial Interview Questions and Answers

D.4 Running Tests

To Run the Tests

1. Change the View Filter setting in the tool bar from All Tests to Last Test Run.

   Choose Last Test Run in the View Filter drop-down menu located in the tool bar. See Figure D-4, for the location of the View Filter combo box. This changes your view of the test tree so that you only see the results of the current test run. This is generally the view that most users prefer. Refer to the JavaTest Harness User's Guide: Graphical User Interface for a detailed description of the JavaTest harness view filters.

   ---

   Note - When you change to the Last Run filter before you do a test run, the folders and tests in the tree turn to gray, indicating that they are filtered out. This occurs because results fo not exist from your last test run.

   ---

2. Choose Run Tests -> Start to start the test run.

   The harness begins to run tests. You can see activity in the test tree panel that indicates the tests currently running. You can also watch the progress of the test run in the progress monitor on the bottom-right portion of the JavaTest harness window and the pie chart in the Summary tab.

3. Expand the test tree folders to reveal the tests.

   Click different test folders to expand the test tree. See Figure D-4 for an example of an expanded test tree.

   Figure D-4 Expanded Test Tree Example

   

D.4.1 JavaTest Harness GUI — Folder and Test Colors

As tests complete, the tests and their folders change color to represent their state.

Table D-2 briefly describes the colors and their meaning.

Table D-2 Folder and Test Colors and Their Meaning

Folders reflect the state of the tests hierarchically beneath them. You know that the entire test suite passed if the test suite root folder is green. See the JavaTest harness online help for more information.

Note - The test lists\DoublyLinkedList\InsertTest.html intentionally contains errors and is supposed to fail as part of the tutorial. If any other tests fail, check your answers to the configuration interview.

D.5 Browsing Test Results

When the test run is complete, use the Folder tabbed pane and Test tabbed pane portion of the JavaTest harness to examine the results. Examine the output of the test that failed.

Note - The Folder tabbed pane and the Test tabbed pane occupy the same portion of the Test Manager window. The Folder tabbed pane is displayed when you choose a folder entry in the test tree and the Test tabbed pane is displayed when you choose a test entry in the test tree.

D.5.1 The Folder Pane

The Folder tabbed pane displays information about the tests in the selected folder. Figure D-5 shows an example of the folder pane.

Figure D-5 Folder Pane Example

Using the Folder Pane

1. Click the top folder in the test tree (the test tree root).
2. Click the Summary tab (shown by default) to display the Summary panel.

   Notice the statistics displayed in the Summary panel. It describes how many tests in the test suite passed, failed, had errors, and were filtered out.

3. Click any of the other folder icons in the test tree to display the statistics for the selected tests.

   This action causes the Summary panel to display the statistics for all tests hierarchically beneath the selected folder.

4. Click the test tree root folder again to redisplay the test suite statistics.

   This action causes the Summary panel to redisplay the statistics for the complete test suite.

5. Click the Passed tab to display a list of tests that passed during the test run.
6. Click the Failed tab to display a list of the tests that failed during the test run (only one test in this case).
7. Double-click the lists\DoublyLinkedList\InsertTest.html test in the Failed tab to change the display from the Folder pane to the Test pane.

   This action selects the test in the test tree and changes the display from the Folder pane to the Test pane.

   ---

   Note - To read more information about any of the panes, click on a tab to establish focus and press F1 to activate online help about that pane.

   ---

D.5.2 The Test Pane

The Test tabbed pane displays information about the selected test. The five tabs provide information about the test and information about the results of its execution. Figure D-6 shows an example of the test pane.

Figure D-6 Test Pane Example

Each tab displays information collected by the JavaTest harness about the test highlighted in the test tree.

Table D-3 briefly describes each tabbed pane.

Table D-3 Test Pane Tabs

Note - To read more information about any of the panes, click on a tab to establish focus, and press F1 to activate the online help about that pane.

Using the Test Pane

1. Click the Test Run Messages tab to view the messages generated by the JavaTest harness or the test during execution.

   Notice that the red icons indicate that the test failed.

2. Click the execute messages item in the left column to display the command line used to run the test.

   The display on the right shows the command line used to run the test. Problems can often be debugged by examining how the test was invoked. In this case it was invoked correctly. See Figure D-7 for an example of the test messages pane.

   Figure D-7 Test Messages Pane Example

   
3. Click the out1 item in the left column to display errors reported by the test.

   The display on the right shows errors reported by the test. The messages indicate that either the test or the API contain errors. In this example, the test contains errors. Figure D-8 shows an example of logged error messages in the test messages pane.

   Figure D-8 Logged Error Messages Example

   

D.6 Excluding a Failed Test

The JavaTest harness allows you to exclude tests from a test suite by specifying an exclude list file. This section shows you how to use the Quick Set Mode of the Configuration Editor to specify an exclude list that includes lists\DoublyLinkedList\InsertTest.html. Tests that are excluded are not executed during test runs, and though they are still displayed in the test tree, their status is not reflected in the pass-fail status of the test suite.

To Exclude a Failed Test

1. Choose Configure -> Change Configuration -> Exclude List from the Test Manager menu bar.

   The Configuration Editor opens directly to a panel that enables you to specify an exclude list. Quick Set Mode allows you to quickly change values that change frequently between test runs. These values are also referred to as standard values. Standard values can also be changed using the Configuration Editor in Question Mode.

2. In the Exclude List pane, click Other.

   This activates a tool used to specify a set of exclude lists.

3. Click the Add button on the upper-right portion of the tool.

   This invokes a file chooser used to specify an exclude list. The current directory of the file chooser is the directory that you were in when starting the JavaTest harness. If it is not, navigate to that directory.

4. Double-click the lib directory entry in the file chooser.
5. Choose the demo.jtx entry in the file chooser and click Select.

   Notice that the exclude list (demo.jtx) is added to the Exclude Lists text box.

6. Click Done in the Configuration Editor.
7. Change the View Filter setting in the tool bar from Last Test Run to Current Configuration.

   Choose Current Configuration in the View Filter located in the tool bar. Figure D-4 displays the location of the View Filter. This changes your view of the test tree so that it displays only the results of the tests selected and filtered out in the configuration. This filter shows only the tests that will run next, as opposed to the Last Test Run filter that shows only the tests that have been run.

   Notice that the icon for the InsertTest.html entry in the Test tree changes from red to gray. This indicates that the test is filtered out and will not be executed. Also notice that the Test Suite Root folder changes from red to green, indicating that all the currently selected tests passed.

D.7 Generating a Report

You can use the JavaTest harness to generate an HTML report that describes the results of the test run. All of the information contained in the report is available from the GUI. The following steps describe how to generate and browse a report of the test run completed in the previous sections of this tutorial.

To Generate a Report

1. Choose Report -> Create Report.

   The Create a New Report dialog box opens.

2. Specify the directory where you want the JavaTest harness to write the report files.

   Use a file chooser to specify the directory and click on the Browse button.

3. Click the Create Report(s) button.

   The reports are generated and you are asked whether you want to view the report.

4. Click Yes.

   The reports are displayed in the JavaTest harness report browser window. Scroll through the report and follow the various links to view data about the test run.

5. To print the report, open the report in your web browser and print it.