This is a specification document, not a tutorial. For more basic
information please consult the jtreg FAQ at http://openjdk.java.net/jtreg.
Contents: Leading comments and defining files ·
Tag syntax · Informational tags · Declarative tags · Action tags (action
options, class names as action
arguments, class name wildcards,
action types) · Order of tags · TestNG
· Directory context ·
Test groups · Source-directory structure and test suite
configuration files · Defaults
· Shorthands · Examples · Supported
names for @requires · Test-specific
system properties and environment variables
A particular test may involve several files of various types,
but the test is specified in just one of them. Let this be called
the defining file of the test.
The defining file of a test may be one of several types of
files. Test tags must be enclosed in a comment at the head of the
file, optionally preceded by comments which do not contain test
tags. The comment in the defining file which includes the test tags
is called the leading comment. The following types of
defining files are recognized:
- A Java source file, with file extension ".java". Only the
"
/*
" to "*/
" comment syntax is
recognized. On each comment line leading asterisks, if any, are
ignored.
- A Bourne shell file, with file extension ".sh". The leading
comment is the first contiguous set of lines each of whose first
character is '#'. The comment is terminated by the first line whose
first character is not '#'. Note that shell tests should not
contain "#!/bin/sh" on the first line; they are intended to be run
only via explicit invocation of a shell.
- An HTML file, with file extension ".html". The leading comment
is enclosed within the usual "
<!--
" to
"-->
" HTML tag.
The contents of a leading comment are parsed into a sequence of
tokens. A token is a maximal contiguous sequence of non-whitespace
characters. Whitespace characters, which include newline and
carriage-return characters, are significant only insofar as they
separate tokens.
Some files within the source directory structure may be
configured to implicitly be defining files. For all other files,
the first token in the leading comment in the defining file of a
test must be "@test
". The harness discovers defining
files by looking for such tokens: a file without comments is not a
defining file; a file whose leading comment does not start with
"@test
" is not a defining file.
TAG SYNTAX
Within a leading comment, a tag consists of a tag token followed
by some number of arguments. A tag token is any token whose first
character is '@'. There is at present no syntax for quoting a
leading '@' character. Case is significant when comparing tag
tokens; all tag tokens defined here are in lower case. It is an
error for the leading comment of a defining file to contain any tag
token not defined here.
As a special case, the token "@(#)
" is not treated
as a tag token. For historical reasons, this token may appear in
the arguments to an @test
tag as part of an SCCS
identification string.
The arguments of a tag are simply the tokens that follow it. The
argument tokens of a tag extend from the first token after the tag
token to the end of the comment, the end of the file, or the next
tag token, whichever comes first. Argument tokens are separated by
whitespace; if commas are used, they will be considered to be part
of the tokens to which they are adjacent.
The arguments of a tag may contain references to various
properties, using the syntax
${<name>}
. The set of names is the
same as the set that may be used in an @requires
expression, and is defined in Appendix
1. Note: For compatibility reasons, this
feature is only enabled by default when the test suite declares a
required version of jtreg 4.2
b14
and higher. The feature can be explicitly enabled or
disabled by setting allowSmartActionArgs
in the
configuration files.
Informational tags document the test. They do not affect the
operation of the harness, but may be used by the harness in
generating reports.
- @test <token>*
-
Defining-file identifier; in general, <token>*
is a comment and is ignored.
As a special case, if the file contains multiple comments
beginning with @test
, if the tokens that follow
@test
are id=
identifier, the
identifier will be used to construct a unique name for the test. If
there is no such identifier, a default one will be used.
- @bug <bugid>+
-
Java bug system numbers
- @summary <token>+
-
Textual summary
- @author <token>+
-
Original author. Use of this tag is now generally discouraged;
information about the author and other contributions is generally
better left to the source code management system for the test
suite.
- @comment <token>*
-
A comment; ignored
Any particular informational tag, except the
@comment
tag, may occur at most once in a given test.
The @comment
tag may be used multiple times.
Declarative tags govern the execution of the tags that follow
them.
- @library <path|jar>+
-
Add one or more pathnames or jar filenames to the library path
list. Each argument must use forward slash to separate file name
components, and may use ".." to denote parent directories.
- If an argument is relative (that is, it does not begin with
'/'), it will be evaluated relative to the directory containing the
test. It is an error if the resulting path is outside the test
suite.
- If an argument begins with '/', it will first be evaluated
relative to the root directory of the test suite. It is an error if
the resulting path is outside the test suite. If the result does
not identify an existing directory, it will be further evaluated
against each entry of a search path in turn, until an existing
directory is found. The search path is specified by the
external.lib.roots
entry in the test suite configuration files.
- An argument may begin with
${java.home}/
to refer
to libraries bundled with the JDK being tested. An argument may
begin with ${jtreg.home}/
to refer to libraries
bundled with the jtreg test harness.
It is an error if any of the entries in the library path list
does not identify an existing directory or jar file.
Each entry in the library path list is categorized according to
its content.
- packages: the contents are one or more source files
arranged in a standard package hierarchy.
- user modules: the contents are one or more directories,
each containing the complete source code for a user module. Each
directory must be named for the module it contains, and the
directory must directly contain a module declaration
(module-info.java).
- system module patch: the contents are one or more
directories, each containing source files to be compiled and used
to patch modules in the system image being tested. Each directory
must be named for the module to be patched, and the directory must
not contain a module declaration (module-info.java or
module-info.class). The source files in each directory must
be organized in a standard package hierarchy.
- precompiled jar file: the entry is a jar file containing
precompiled class files.
It is an error if any entry in the library path list cannot be
categorized. In particular, any one entry in the library path
cannot contain more than one type of content. It is highly
recommended that a library directory should only contain content
directly related to the library; it should not contain
subdirectories containing other content, such as tests, or other
separate libraries.
In general, classes in library directories are not automatically
compiled as part of a compilation command explicitly naming the
source files containing those classes. A test that relies upon
library classes should contain appropriate @build
directives to ensure that the classes will be compiled. It is
strongly recommended that tests do not rely on the use of implicit
compilation by the Java compiler. Such an approach is generally
fragile, and may lead to incomplete recompilation when a test or
library code has been modified.
If a library directory is used by a directory of TestNG tests, all the classes in a library directory
will be automatically compiled by means of implicit
@build
tags.
When the harness compiles a test, the source code for each entry
(if any) will be placed on the appropriate source path, and any
previously compiled classes for each entry will be placed on the
class path, application module path or module patch path as
appropriate. When the harness runs a test, the compiled classes for
each entry will be placed on the class path, application module
path or module patch path as appropriate.
The @library
tag may be used more than once. It may
only be used before the first @run
tag (either
explicit or implicit.) Specifically, it must appear before any use
of @run
, @build
, @compile
,
and @ignore
.
The @library
tag and the library path list have
nothing to do with the search path (i.e., the PATH environment
variable) defined in shell actions.
- @key <keyword>+
-
Label this test with the given keyword(s). Some harnesses allow
tests to be selected via boolean expressions on keywords. The list
of acceptable keywords may be specified in the TEST.ROOT file (see below). The @key
tag may be used at most once in a given test.
- @modules
<module>[/<package>[:<flag>]]+
-
Express a dependence on modules in the system being tested, and
optionally, on selected internal packages in some or all of those
modules.
If no dependencies are specified explicitly, a default set of
dependencies will be read from the modules
entry in
test suite configuration files.
The dependencies will be ignored if the version of JDK used to
run the tests does not support the use of modules. Otherwise, a
test will not be run if the system being tested does not contain
all of the specified modules. All of the specified modules will be
present in the configuration when the test is run.
A flag may be specified to indicate the desired access to a
specified package.
Flag |
Access |
none |
The package should be exported, so that its public types will
be available for use by the test at compile time and run time. |
open |
The package should be opened, so that its types will be
available for use by the test at run time using deep
reflection. |
+open |
The package should be both exported and opened, so that its
public types will be available for use by the test at compile time
and run time, and other types will be available at run time using
deep reflection. |
- @requires <expression>
-
Express a dependence on characteristics of the system being
tested. Some harnesses allow tests to be selected according to the
characteristics of the system being tested. The expression may be
composed of the following elements:
- Named characteristics. See Appendix
1 for a complete list of the standard supported names, and
details of the test-suite configuration files
for information on how to extend the set with additional names
specific to the test-suite.
- Numeric values. Values must be decimal integers, optionally
followed a suffix '
K
', 'M
' or
'G
', which as a a multiplier of 1024, 1024*1024 and
1024*1024*1024 respectively. The resulting value must fit in a
long
variable.
- String values, enclosed in double quotes '"'.
- Boolean values '
true
' and
'false
'.
- Operators:
( )
*
/
%
+
-
==
~=
<
<=
>
>=
&
|
!
The @requires
tag may be used multiple times in a
given test. If it is used more than once, the expressions in the
individual instances will be enclosed in parentheses and combined
with '&'.
Action tags tell the harness how to perform the test. They are
executed in the order in which they are given. Each action either
passes or fails. If an action fails, no following actions, if any,
are performed. A test passes if, and only if, all of its actions
pass.
Action tags begin with the tag token "@run
", and
have the following syntax:
@run <type><option>* <arg>*
The <type> describes the basic type of the
action; the <option>s further describe how the
action is to be performed. The <arg>s are passed
to the test in a manner appropriate to the
<type>.
The first token after the "@run
" token contains
both the action type and options, if any. To parse the type and the
options, this token is broken down into subtokens. A subtoken is
either the character '=', the character '/', or a maximal
contiguous sequence of characters not containing either of these
characters.
The type of an action tag is named by the first subtoken of the
tag's first token. The remaining subtokens of the first token form
the options. Options have the syntax
/<name>
or
/<name>=<value>
where <name> and <value> are a
single subtokens. The <value> may be enclosed in
double quotes to prevent the usual interpretation of '=', '/', and
whitespace.
ACTION OPTIONS
Not all action types support all options.
- /fail
-
Negate the result of the action. If the action fails it is
treated as though it passed, and vice versa.
- /timeout=<seconds>
-
Specify the timeout value. The default timeout is two minutes.
If an action does not finish before the timeout expires, it fails.
The timeout period applies to the entire action, not to particular
steps of the action. A value of zero means there is no timeout. The
/timeout option may not be given in conjunction with the
/manual
option.
- /ref=<file>
-
Capture the standard output and error of the action and compare
it to the content of <file>, which is in the same
directory as the test. The action succeeds only if the output
matches the content of the file. Even if /fail
is
specified, the output must match in order for the action to
pass.
- /othervm
-
Some test harnesses reuse a VM between tests. This option forces
the action to be run in a fresh VM subprocess. Use this option if
you need to specify VM options, or if the action might crash the
VM.
- /manual[=(yesno|done)]
-
Indicates that this is a manual action requiring user
interaction. If the harness has been instructed to run only
automatic actions, then this action will be skipped and will be
considered to have passed. The /manual
option may not
be given in conjunction with the /timeout
option.
If no option value is given, then the harness assumes that the
test itself will handle whatever user interaction is necessary. If
"yesno
" is given, then the harness will ask the user
whether the action is to pass or fail. If "done
" is
given, then the harness will wait until the user indicates that the
test has completed. When the harness queries the user it does so in
a manner appropriate to the action type; e.g., for applet actions
it will either display "pass
" or "fail
"
buttons or a single "done
" button. In the failure
case, some harnesses may provide a way for the user to submit text
describing the failure.
- /policy=<file>
-
Define the Java system property "java.security.policy" to have
the value "=${TESTSRC}/
<file>", where
TESTSRC is the name of the directory containing the defining file
of the test. This definition has the effect of making the given
file the sole source of security policies to be used by the
security manager. In particular, the system security policy and the
user's security policy will be ignored. If the /secure
option is not used then the default security manager will be
installed.
Property expansion will be performed in the policy file as
usual; in particular, the "test.src
" and
"test.classes
" properties will name the source and
class directories of the test being run (see DIRECTORY CONTEXT below).
This option may only be used in conjunction with the
/othervm
option.
- /secure=<class>
-
Specify a subclass of java.lang.SecurityManager
to
be installed as the security manager. An appropriate
@build
tag should be provided to ensure that the class
is compiled. If the /secure
option is used without the
/policy
option then the system's built-in policy,
equivalent to the original sandbox policy, will be assumed.
This option may only be used in conjunction with the
/othervm
option.
- /native
-
The test uses native code, which must be provided separately, in
a precompiled library, when the test is run.
CLASS NAMES AS ACTION ARGUMENTS
Some actions take one or more class names as arguments. Each
such argument is the name of a class, not the name of a class file,
so the ".class" suffix should not be given.
A class name may be specified in one of the following ways:
- a simple class name: e.g., "Foo"
- a package name followed by a simple name: e.g., "bar.Foo",
"baz.bar.Foo"
- a module name followed by either
module-info
or a
fully qualified class name: e.g., "myModule/module-info",
"myModule/baz.bar.Foo"
If a class in a non-Java package is specified, then the source file
for that class must be in the corresponding directory relative to
the directory containing the defining file of the test.
CLASS-NAME WILDCARDS
Some actions take either class names or class-name wildcards as
arguments. A class-name wildcard can be specified in one of the
following ways:
- "*", meaning all classes in the default package
- "<package-name>.*", meaning all classes in the
named package
-
"<module-name>/<package-name>.*",
meaning all classes in the named package in the named module
- "<module-name>/*", meaning all classes in the
named module
ACTION TYPES
- build <classname-or-wildcard>+
-
Build classes on demand. Each argument may be a class name, as
described above, or a class-name wildcard. To locate a source file
for a named class, the harness takes the first matching file found
by looking in the test-source directory and then in each
appropriate directory of the library path list. To locate the
source files denoted by a wildcard, the harness evaluates the
wildcard relative to test-source directory and relative to each
directory of the library path list, taking all matches. Each
discovered class will be compiled if its corresponding class file
does not exist or is older than its source file. Intended primarily
for use before main and applet actions that require more than one
class to be compiled. Passes only if the compiler finishes without
error, or if none of the specified classes need to be compiled.
It is an error for a build action to be the last action in a
test description.
- clean <classname-or-wildcard>+
-
Remove the class files for the named classes, if they exist.
Each argument may be a class name or a class-name wildcard.
Intended primarily for compiler tests. Always passes.
-
compile[/fail][/ref=<file>][/timeout=<seconds>][/process][/module=<module-name>]
<arg>+
-
Invoke the compiler on the given <arg>s, which
may include any compiler option other than the "-d
"
option. Equivalent to "javac <arg>+", therefore to
specify source files the ".java" suffix must be included. Source
file names should be relative pathnames using forward slashes and
".." to denote parent directories; they are normally interpreted
relative to the directory containing the defining file of the test.
Standard output and standard error are concatenated (not
interleaved) so that /ref
may be used. Intended
primarily for compiler tests; non-compiler tests should generally
use build actions. The timeout period, if specified, applies to the
entire compile action, not to the compilation of each individual
source file. Passes only if the compiler finishes without
error.
The /process
option can be given to indicate that
the compilation includes the use of an annotation processor and
that no source files are being compiled. If /process
is not given, at least one source file must be specified.
The /module=<module-name>
option
is used to compile a patch for a module in the system image. In
this one situation, the source files must be in a subdirectory
named for the module, relative to the directory containing the
defining file of the test. Neither the module directory nor the
list of source files may contain a module-info.java
file. Internally, this option corresponds to using the javac
--patch-module=<module-name>=<path>
option.
User modules can be compiled by placing the code for the modules
in subdirectories named for the modules. Each directory should have
a corresponding module-info.java
file.No other options
are required: in particular,
/module=<module-name>
should
not be used; that option is just used for patches to system
modules. To avoid confusion, it is highly recommended that tests
containing user modules should be placed in their own directory,
and should not contain any other directories apart from the
directories used to contain module source code.
The use of the -classpath
and
-sourcepath
options as arguments to the compile action
is discouraged; the @library
tag is often more
appropriate.
-
main[/fail][/manual][/othervm][/policy=<file>][/secure=<class>]
[/timeout=<seconds>] <vm-opt>*
<class> <arg>*
-
Invoke the main method of the specified class, passing any
arguments after the class name. Roughly equivalent to "java
<vm-opt>* <class>
<arg>*". All initial argument tokens beginning
with a dash are considered to be VM options; the first argument not
beginning with a dash is the <class> to be
invoked, and any remaining arguments are passed to the main method
of that class. Passes only if the main method returns normally and
does not cause an exception to be thrown, from the main thread or
from any other thread. A "main" action is considered to be finished
when the main method returns; if a test involves multiple threads,
some synchronization may be necessary to ensure that the other
threads finish their work before the thread running the main method
returns.
The named <class> will be compiled on demand,
just as though an "@run
build
<class>" action had been inserted before this
action. If this action requires classes other than
<class> to be up to date, insert an appropriate
build action before this action. If any <vm-opt>s
are given then /othervm
should be specified. If the
/manual
option is specified then the harness will skip
this action when instructed to run only automatic tests; no
arguments to the /manual
option are supported.
The use of the -classpath
option as an argument to
a main/othervm
action is discouraged; the
@library
tag is often more appropriate. The setting of
the system properties "java.security.manager
" and
"java.security.policy
" in main/othervm
actions is also discouraged; the /policy
and
/secure
options should be used instead.
- driver[/fail][/timeout=<seconds>]
<class> <arg>*
-
Invoke the main method of the specified class, passing any
arguments after the class name. Although superficially similar to
@run main
, this is for use when it is desirable to
perform additional setup or configuration before running the class
containing the actual test code, possibly in a child VM.
The named <class> will be compiled on demand,
just as though an "@run
build
<class>" action had been inserted before this
action. If this action requires classes other than
<class> to be up to date, insert an appropriate
build action before this action.
-
junit[/fail][/manual][/othervm][/policy=<file>][/secure=<class>]
[/timeout=<seconds>] <vm-opt>*
<class>
-
Invoke the specified class as a JUnit 4.5 test. Roughly
equivalent to "@run main org.junit.runner.JUnitCore
<class>
". All initial argument tokens
beginning with a dash are considered to be VM options; the first
argument not beginning with a dash is the <class>
to be invoked. No additional arguments are allowed. Passes only if
the main method of JUnitCore returns normally and does not cause an
exception to be thrown, from the main thread or from any other
thread. As with a "main" action, a "junit" action is considered to
be finished when the main method of JUnitCore returns.
The named <class> will be compiled on demand,
just as though an "@run
build
<class>" action had been inserted before this
action. If this action requires classes other than
<class> to be up to date, insert an appropriate
build action before this action. If any <vm-opt>s
are given then /othervm
should be specified.
The use of the @library
tag, and of the
/othervm
, /manual
, /policy
,
and /secure
options, are the same for a "junit" action
as for a "main" action; see above.
In order to use a "junit" action, the JUnit 4.5 jar file may
need to be placed manually alongside the jtreg and javatest jar
files in the jtreg installation. If junit.jar file is not found,
and if no class named org.junit.runner.JUnitCore
can
be found, any "junit" actions will fail, regardless of whether the
/fail
option was selected.
This action executes all the tests in a single JUnit test file.
All the tests in that one file will be grouped as a single jtreg
test for reporting purposes.
-
testng[/fail][/manual][/othervm][/policy=<file>][/secure=<class>]
[/timeout=<seconds>] <vm-opt>*
<class>
-
Invoke the specified class as a TestNG test. Roughly equivalent
to "@run main org.testng.TestNG
<class>
". All initial argument tokens
beginning with a dash are considered to be VM options; the first
argument not beginning with a dash is the <class>
to be invoked. No additional arguments are allowed. Passes only if
the main method of TestNG returns normally and does not cause an
exception to be thrown, from the main thread or from any other
thread. As with a "main" action, a "testng" action is considered to
be finished when the main method of TestNG returns.
The named <class> will be compiled on demand,
just as though an "@run
build
<class>" action had been inserted before this
action. If this action requires classes other than
<class> to be up to date, insert an appropriate
build action before this action. If any <vm-opt>s
are given then /othervm
should be specified.
The use of the @library
tag, and of the
/othervm
, /manual
, /policy
,
and /secure
options, are the same for a "testng"
action as for a "main" action; see above.
In order to use a "testng" action, the TestNG jar file may need
to be placed manually alongside the jtreg and javatest jar files in
the jtreg installation. If testng.jar file is not found, and if no
class named org.testng.TestNG
can be found, any
"testng" actions will fail, regardless of whether the
/fail
option was selected.
This action executes all the tests in a single TestNG test file.
All the tests in that one file will be grouped as a single jtreg
test for reporting purposes.
Note: this action allows individual tests to be written as
TestNG tests, and placed alongside other tests using other
combinations of actions. See below for
details on how to set up a directory containing only standard
TestNG tests.
-
applet[/fail][/manual[=(yesno|done)]][/othervm][/policy=<file>]
[/secure=<class>][/timeout=<sec>]
<html-file>
-
Run the applet described by the first
<applet>
HTML tag in the given
<html-file>, ignoring any other
<applet>
tags. The applet action is roughly
equivalent to "appletviewer <html-file>"; the
major difference is that the applet is not run in the restricted
security environment implemented by the appletviewer. Intended
primarily for tests of graphics functionality; tests of the
appletviewer itself can be written using shell actions. Passes if
the applet does not throw any exceptions from any thread and, if
the /manual
option is specified with a value of
"yesno
", the user indicates that the test passes.
The class named in the <applet>
tag will be
compiled on demand, just as though an "@run
build
<class>" action had been inserted before this
action. If this action requires classes other the class named in
the <applet>
tag to be up to date, insert an
appropriate build action before this action.
If /manual
is not specified, then the applet will
be run by invoking its init
, start
, and
setVisible(true)
methods (in that order), delaying for
a few seconds, and then invoking its stop and destroy methods (in
that order).
The /manual
option may be specified to indicate
that the test requires human interaction. In this case the HTML
file itself will be displayed; it should contain any instructions
for the user. When displaying the HTML file, a harness may or may
not interpret HTML tags other than the <applet>
tag; thus the instructions should be written in a way that does not
depend upon HTML rendering.
If /manual
is specified alone, i.e., without any
value argument, then the applet will be run just as in the
non-manual case. The harness will skip this action, and treat it as
though it had passed, when instructed to run only automatic
tests.
If /manual=yesno
is specified, then the harness
will ask the user whether the test passes or fails, typically by
displaying "pass
" and "fail
" buttons. The
applet will be run by invoking its init
,
start
, and setVisible(true)
methods (in
that order), waiting for the user to click on "pass
"
or "fail
", and then invoking the applet's
stop
and destroy
methods (in that
order).
If /manual=done
is specified, then the harness will
wait for the user to indicate that the test is complete, typically
by displaying a "done
" button. The applet will be run
by invoking its init
, start
, and
setVisible(true)
methods (in that order), waiting for
the user to click on "done
", and then invoking the
applet's stop
and destroy
methods (in
that order).
The setting of the system properties
"java.security.manager
" and
"java.security.policy
" in applet/othervm actions is
also discouraged; the /policy
and /secure
options should be used instead.
- shell[/fail][/manual][/timeout=<seconds>]
<script> <arg>*
-
Invoke the Bourne shell to run <script> on the
given <arg>s. The <script>
argument should the name of a file relative to the test source
directory (see below), using "/" to separate directories and ".."
to refer to parent directories. Passes only if the script exits
with an exit status of 0.
On Unix systems, the PATH
will be set to
/bin:/usr/bin:/usr/sbin
and the following variables
will be propagated if they are set: DISPLAY
,
GNOME_DESKTOP_SESSION_ID
, HOME
,
LANG
, LC_ALL
, LC_CTYPE
,
LPDEST
, PRINTER
, TZ
and
XMODIFIERS
.
On Windows systems, the following variables will be propagated
if they are set: PATH
, SystemDrive
,
SystemRoot
, TEMP
, TMP
,
TZ
and windir
. In addition, when using
Windows Subsystem for Linux (WSL) to run shell tests:
WSLENV
will be set appropriately; and
EXE_SUFFIX
will be set to .exe
, to
facilitate creating paths for tools to be executed by the shell
script.
In addition, a number of test-specific environment variables
will be set, giving details about the test itself. See Appendix 2 for the complete list of these
variables.
If the /manual
option is specified then the harness
will skip this action when instructed to run only automatic tests;
no arguments to the /manual
option are supported.
- ignore <word>*
-
Ignore this and all following @run
tags. A test
harness may treat this test as a failure or as some other type of
error. The <word> tokens, if any, should describe
why the test is being ignored. These tokens may be displayed by the
harness in some appropriate fashion.
ORDER OF TAGS
The following is the recommended order for the tags in a test
description:
@test
@bug
@summary
@key
@requires
@modules
@library
- action tags, in the order to be executed
TESTNG
A directory within the source directory structure may be
configured to be the root of a set of TestNG tests. When so
configured, all Java source files in or under the directory are
considered to be defining files, and thus considered to be a test
to be executed. If a file has a leading comment, the comment may
only contain informative tags and declarative tags, and may not
contain any action tags.
Each test within the set is deemed to have any implicit
@build
action specifying all the Java source files in
the set, followed by an implicit action to execute the test as a
TestNG test.
DIRECTORY CONTEXT
Each test is run in a context that defines the following
directories.
- Working directory
- The directory in which the harness is running.
- Source directory
- The directory containing the defining file of the test, as well
as any associated files (e.g., input data files). Generally not the
same as the working directory.
- Class directory
- The directory into which class files compiled from source files
in the source directory are compiled. May be the source directory,
or may be some other directory.
- Java home directory
- The directory containing the JDK build or release being
tested.
A test may create temporary files in the working directory as
needed. Test harnesses will typically delete such files before or
after each test is run. In order to work properly when run
standalone, however, tests should not rely upon this behavior. This
can be done having each test's initialization code delete any files
that the test may have created in a previous invocation.
For each source directory in the test hierarchy there is a
corresponding class directory, which may or may not the same as the
source directory. Classes in different source directories are,
therefore, in different name spaces and their names will not
collide. Simple test harnesses may place class files in source
directories, while more sophisticated harnesses will generally
place class files in a parallel directory hierarchy.
The names of the source, class, and Java home directories of a
test are made available to shell-action scripts via the environment
variables TESTSRC
, TESTCLASSES
, and
TESTJAVA
, respectively. The directory names do not
have trailing separators.
The names of the source and class directories of a test are made
available to main and applet actions via the system properties
"test.src" and "test.classes", respectively. The directory names do
not have trailing separators. Main and applet actions can read data
files defined in the test's source directory using this idiom:
File f = new File(System.getProperty("test.src", "."), "foo");
InputStream in = new FileInputStream(f);
By defaulting the directory to ".", this technique allows the
test to be run standalone in its source directory.
TEST GROUPS
Groups are defined in a test suite using one or more Java
properties files. The names of these files must be listed in the
"groups
" entry in TEST.ROOT.
If the filename is enclosed in square brackets, no error message
will be given if the file cannot be found. Within the property
files, each entry specifies items to be included or excluded from
the group.
- To include a test or directory of tests, simply specify the
name of the test or directory.
- To exclude a test or directory of tests, use '-' followed by
the name of the test or directory. There must be no spaces between
the "-" and the name that follows.
- To include the contents of another group, use ':' followed by
the name of the group. There must be no spaces between the ":" and
the name that follows.
SOURCE-DIRECTORY STRUCTURE AND TEST-SUITE CONFIGURATION FILES
This specification places few constraints upon the structure of
a tree of source directories. Some test harnesses may be able to
provide more robust behavior or more succinct reports when they can
identify the root of such a tree. The root of a test-source tree
must therefore be identified by placing a file named TEST.ROOT in
the root directory. Exactly one such file must exist in every
test-source tree.
The TEST.ROOT file contains
test-suite configuration information. It is in the standard Java
property-file format as described in the specification of the
java.util.Properties
class. Properties defined in this
file apply to the entire test suite. Additional properties files
named TEST.properties may appear in subdirectories of the test
suite, and may be used to provide additional configuration values
that apply to files in that directory and its subdirectories.
The following properties may only appear in TEST.ROOT:
checkBugId true|false
-
Whether or not to check the format of strings in @bug tags.
Checking is disabled if this property is set to
false
.
defaultExecMode
agentvm|othervm|samevm
-
The default execution mode for tests in the test suite. A
harness that supports multiple test execution modes may use this
value to determine the default execution mode for tests in this
test-suite.
groups <files>
-
Where to find group definition files. A space separated list of
files. Optional files should be enclosed in square brackets.
requiredVersion
<version>
-
Minimal version of the harness required for this test-suite. A
version string in the format "[0-9.]+ ?b[0-9]+" (without the
quotes). For example a value of "4.1 b03" declares that at least
that version of the harness is required to run all the tests in the
test suite.
-
This option is used to provide source files for classes that
will be used at the beginning of each test suite run, to determine
additional characteristics of the system for use with the
@requires
tag. Each class must implement
java.util.concurrent.Callable<java.util.Map<String,
String>>
. When invoked, the call()
method
should return properties that can be referenced in an expression in
a @requires
tag. Note: the names of the new
properties that may be returned by calling these classes must also
be declared in a requires.properties
entry. If an
error occurs while computing a value, the value may be set to the
special value __ERROR__
, optionally followed by a
short explanation of the error that occurred.
If this option is specified, the following additional options
may also be specified:
requires.extraPropDefns.libs
<source-files>
— source files for
classes that will be put on the classpath when the primary classes
are run.
requires.extraPropDefns.bootlibs
<source-files>
— source files for
classes that will be put on the bootclasspath when the primary
classes are run.
requires.extraPropDefns.javacOpts
<options>
— options that will be
passed to javac when the source files are compiled.
requires.extraPropDefns.vmOpts
<options>
— options that will be
passed to VM when the classes are executed.
In this family of options, if a source file is enclosed in
square brackets, no error message will be given if the file is not
available.
The following properties may appear in either TEST.ROOT or any
TEST.properties file:
keys <keywords>
-
The set of valid keywords that can be used in @key tags. A
harness that supports keyword-based test selection should reject
any test containing keywords that are not present in this list. If
this property is not present then no keywords should be
accepted.
bootclasspath.dirs
<directories>
-
Directories that when used should be placed on the
bootclasspath.
othervm.dirs <directories>
-
Directories containing tests which should all be run in othervm
mode. A harness that supports multiple test execution modes should
ensure that all tests in the specified directories are run in
othervm
mode.
exclusiveAccess.dirs
<directories>
-
Directories containing tests which should not be run
concurrently at the same time as other tests. A harness that
supports concurrent test execution should ensure that all tests in
the specified directories are run when no other tests in the same
directory is being run.
TestNG.dirs
<directories>
-
Directories containing TestNG-style tests.
lib.dirs
<directories>
-
Default library directories used by all tests in or under the
directory containing this property declaration.
lib.build
<build-arguments>
-
Default library components that will be built by tests in TestNG
directories in or under the directory containing this property
declaration. The values should be of a form that can be accepted by
an @build
tag. If not specified, the entire contents
of any currently specified libraries will be built.
external.lib.roots
<directories>
-
A search path for libraries specified in an
@library
tag. Relative paths will be evaluated
relative to the directory containing the configuration file.
modules
<module[/package[:flag]]>+
-
Express a default dependence, for tests in this directory and
its subdirectories, on modules in the system being tested, and
optionally, on selected internal packages in some or all of those
modules.
requires.properties
<property-name>+
- Declare property names that may be used in a
@requires
expression. The names may identify either
system properties or additional test-suite specific properties
return by calling the classes defined by the
@requires.extraPropDefns
family of configuration
values.
maxOutputSize
<int-value>+
- Override the default limit for the amount of output that will
be saved on any stream being recorded by the test harness. For
jtreg, the default limit for all tests is 100,000 characters, or
can be set on the command line by setting the system property
javatest.maxOutputSize
.
Note:To set a system property
for jtreg on the command line, use
-J-D<name>=<value>
.
allowSmartActionArgs
<true|false>+
- Specify whether or not to permit the use of
${<name>
in the arguments for an
action. The set of names is the same as the set that may be used in
an @requires
expression. See Appendix 1.
DEFAULTS
If no @run
tags are present in a defining file, a
default is assumed depending upon the file's filename
extension.
Default Actions
File type |
Default |
Notes |
.java |
@run main <name> |
<name> is the name of the file without the
".java" suffix |
.sh |
@run shell <file> |
|
.html |
@run applet <file> |
|
SHORTHANDS
Shorthands
Shorthand |
Equivalent |
@build <classname>+ |
@run build <classname>+ |
@clean <classname>+ |
@run clean <classname>+ |
@compile<option>* <arg>+ |
@run compile<option>*
<arg>+ |
@ignore <word>* |
@run ignore <word>* |
EXAMPLES
Run Foo
in a separate VM, with the heap limited to
2MB and the verifier turned on:
/* @test
* @run main/othervm -mx2m -verify Foo arg1 arg2
*/
Run Foo
in the same VM, with a two-second
timeout:
/* @test
* @run main/timeout=2 Foo
*/
Compile Foo
with debugging, expecting failure, and
check the error message against a reference file:
/* @test
@compile/fail/ref=Foo.ref -debug Foo.java
*/
Remove Bar
's class file, if it exists, then compile
Foo
, then compile Bar
, and finally run
Bar
, which is expected to fail:
/* @test
@clean Bar
@compile Foo.java
@compile Bar.java
@run main/fail Bar
*/
Run the applet described in the file Foo.html, waiting for the
user to indicate success or failure.
/* @test
* @run applet/manual=yesno Foo.html
*/
Run the class Snidely
with the security manager
Pinkerton
, using the paranoid.sp policy file:
/* @test
@build Pinkerton
@run main/othervm/secure=Pinkerton/policy=paranoid.sp Snidely
*/
Run the class SnowWhite
using classes in the
library directory dwarfs:
/* @test
@library ../dwarfs
@build Bashful Doc Dopey Grumpy Happy Sleepy Sneezy
@run main SnowWhite
*/
APPENDIX 1: SUPPORTED
NAMES FOR @requires AND ACTION ARGUMENTS
Note: See the details for the test-suite configuration files for information on how to extend
the set of supported names for a particular test suite.
Supported names for @requires
and action
arguments
Name |
Description |
Value if set |
Default value
if not null |
jdk.version |
The JDK version, as given by
java.specification.version system property of the test
JDK. |
|
|
jdk.version.major |
The major component of the version string, i.e. ignoring any
leading "1." for versions prior to JDK 9. |
|
|
os.name |
The operating system name, as given by the corresponding system
property. |
|
|
os.family |
The operation system family, derived from the operating sytsem
name. |
One of linux mac solaris
windows |
first word of os.name |
os.arch |
The operating system architecture, as given by the
corresponding system property. |
|
|
os.simpleArch |
A simplified representation of the operating system
architecture. |
One of x64 i586 ppc |
same as os.arch |
os.version |
The operating system version, as given by the corresponding
system property. |
|
|
os.simpleVersion |
A simplified representation of the operating system
version. |
N.0 or N.M ,
where N and M are first two
numbers in os.version |
99.99 |
os.processors |
The number of processors on this system |
The value as determined by
Runtime::availableProcessors |
|
os.maxMemory |
The maximum amount of memory available on this system. |
The value as determined by
OperatingSystemMXBean::getTotalPhysicalMemorySize |
|
os.maxSwap |
The maximum amount of swap space available on this system. |
The value as determined by
OperatingSystemMXBean::getTotalSwapSpaceSize |
|
vm.gc |
The garbage collection strategy, derived from the option
-XX:+UsenameGC |
name |
|
vm.opt.switch |
A boolean VM option, derived from option
-XX:+switch or
-XX:-switch |
true false |
|
vm.opt.name |
A VM option, derived from option
-XX:name=value |
value |
|
profile |
The name of the highest profile supported by the test JDK. |
One of compact1 compact2
compact3 |
|
module:name |
Whether or not a module is available in the test JDK. |
One of true false |
|
APPENDIX 2: TEST-SPECIFIC SYSTEM
PROPERTIES and ENVIRONMENT VARIABLES
The system properties in the following table will be set while
executing any class specified in an action tag; the environment
variables will be set while executing a shell action tag.
Test-specific system properties and environment
variables
System property |
Environment variable |
Notes |
test.file |
TESTFILE |
The defining file of the test |
test.src |
TESTSRC |
The directory containing the defining file of the test |
test.src.path |
TESTSRCPATH |
The series of directories, including library directories, for
the source of the test. |
test.classes |
TESTCLASSES |
The directory containing the compiled classes for source files
in the directory containing the defining file of the test. |
test.class.path |
TESTCLASSPATH |
The series of directories for all the classes for the
test. |
test.vm.opts |
TESTVMOPTS |
Additional VM options for all JVMs used in the test. |
test.tool.vm.opts |
TESTTOOLVMOPTS |
Additional VM options for all JVMs used in the test, each
preceded by '-J ', suitable for use in tool commands
like "javac". |
test.compiler.opts |
TESTJAVACOPTS |
Additional Java compiler options. |
test.java.opts |
TESTJAVAOPTS |
Additional VM options for all invocations of the main Java
launcher used in the test. (i.e.excluding JDK tools like
"javac".) |
test.jdk |
TESTJAVA |
The location of the JDK being used to run the tests. |
compile.jdk |
COMPILEJAVA |
The location of the JDK being used to compile the tests. By
default, this is the same as test.jdk and
TESTJAVA . |
test.timeout.factor |
TESTTIMEOUTFACTOR |
The timeout factor to be applied to the default timeout for the
test. |
test.nativepath |
TESTNATIVEPATH |
The location of native executables used in the tests. |
test.modules |
TESTMODULES |
The module dependencies of the test, if any, defined in the
@modules tag. The dependencies will be ignored, and
the system property and environment variable will be unset or
empty, if the version of JDK used to run the tests does not support
modules. |
test.root |
TESTROOT |
The root directory of the test suite (the directory containing
the TEST.ROOT file.) |
Not Applicable |
FS |
The file separator character to use. (\ for
Windows tools; / otherwise.) |
Not Applicable |
PS |
The path separator character to use. (; for
Windows tools; : otherwise.) |
Not Applicable |
NULL |
The "null sink" to use. (NUL in a WIndows
environment; /dev/null otherwise.) |