JEP draft: Enhanced javadoc support for code samples (snippets)

OwnerPavel Rappo
TypeFeature
ScopeJDK
StatusDraft
Componenttools / javadoc(tool)
Created2018/04/13 10:54
Updated2021/01/27 00:23
Issue8201533

Summary

Provide enhanced support for including fragments of source code ("snippets") in API documentation generated by javadoc's standard doclet.

Goals

Non-Goals

Success Metrics

Replace most if not all use of <pre>{@code ...}</pre> blocks in key modules in JDK API documentation with instances of a new tag.

Motivation

Authors of API documentation frequently want to include fragments of source code in documentation comments. Although {@code ...} can be used by itself for very small fragments of code, for anything non-trivial, fragments are typically represented in documentation comments with the following compound pattern:

<pre>{@code
    lines of source code
}</pre>

There are various shortcomings to this approach:

A better methodology to address all these concerns is to provide a new tag with metadata that allows the author to implicitly or explicitly specify the kind of content, in order that it can be validated, and presented in the desired manner. In addition, it is desirable to allow the fragments to be placed in separate files that can be directly manipulated in an appropriate manner in the author's preferred editor.

Description

A new inline tag {@snippet ...} is introduced, to declare code fragments to appear in the generated documentation. It can be used to provide inline snippets, where the code fragment is included within the tag itself, and external snippets, where the code fragment is read from a separate, external source file.

For example,

/**
     * The following code shows how to use Optional.isPresent:
     * {@snippet :
     *     if (v.isPresent()) {
     *         System.out.println("v: " + v.get());
     *     }
     * }
     */

or,

/**
     * The following code shows how to use Optional.isPresent:
     * {@snippet file="ShowOptional.java" region="example"}
     */

where ShowOptional.java is a file that can be read by the standard doclet, and which contains a region of text identified by the name example. For example,

public class ShowOptional {
    void show(Optional<String> v) {
        // @start example
        if (v.isPresent()) {
            System.out.println("v: " + v.get());
        }
        // @end
    }
}

A hybrid form is also supported where the code fragment is in an external file, but an equivalent copy of the text can also be provided within the tag, for the convenience of anyone reading the source code for the class being documented.

The complete form of the new tag is as follows:

{@snippet [name=value ...] :
    lines of source code
}

The sequence colon newline and the following lines of source code can be omitted if not required: that is, for an external snippet.

The name=value pairs can be used to specify properties of the code fragment. Values can be quoted with either ' or " if they contain whitespace. No other escapes are supported. The initial set of supported names is as follows:

An external file may contain more than one snippet, to be included at different places within the documentation.

Indentation

When the source code fragment is included in the snippet tag, leading whitespace is stripped from the code fragment using String.stripLeading. This addresses an annoying shortcoming of <pre>{@code ...}</pre> blocks, where the text to be displayed starts immediately after any leading space and asterisk characters.

Markup

It is often desirable to include pseudo-code in code fragments in documentation, or to highlight specific parts of the fragment, even though it is not possible to do so directly in the underlying language of the fragment. To that end, the source code defined by the snippet tag may be marked up with "markup tags" in comments to be interpreted by javadoc when processing the tag, in order to affect the presentation of the text in the generated documentation. These comments use "end-of-line"-style comments appropriate for the kind of content and can be used equally within inline and external snippets. The tags can be used to define subsequences of the text and actions to be performed on those subsequences.

Each subsequence may be either a part of a line, defined by a literal string or regular expression, or a group of lines, defined by @start and @end tags.

The basic set of actions includes:

Markup tags appear in markup comments, which are end-of-line comments beginning with a valid markup tag. A markup comment may contain additional markup tags after the initial tag. Markup comments need not appear on a line by themselves, and may appear after any preceding text on the same line. This allows markup comments to be placed in the source code without affecting line or column numbers of the primary content in the source code.

Markup comments are not part of the generated documentation.

For example, to highlight an entire region, combine the @start [name] and @highlight tag on the same line:

class HelloWorld {
    // @start @highlight
    public static void main(String ... args) {
        System.out.println("Hello World!");
    }
    // @end
}

To highlight part of a line, even in a comment:

class HelloWorld {
    public static void main(String ... args) {
        // write the standard output           // @highlight "standard"
        System.out.println("Hello World!");    // @highlight /".*"/
    }
}

The initial set of markup tags is as follows:

API

The Compiler Tree API will be extended to provide support for the new snippet tag. This will allow external tools to scan the documentation comments in a library for uses of snippet tags, in order for those tools to validate the content of the snippet.

Validating snippets

By providing an API to access the structured content of snippets, we do not constrain the concept of validation to support within the javadoc tool.

A significant advantage of using external snippets is that it is expected that such files will be compilable, in some suitable compilation context. It will be up to the test infrastructure for a library to locate these files, and to verify that they can be suitably compiled, perhaps using the Java Compiler API, and possibly executed as well, perhaps in some appropriate test infrastructure.

For inline snippets, especially those that are not a full compilation unit, it will be up to the test infrastructure to "wrap" the code fragment in a full compilation unit, such that it can be compiled and possibly executed.

For validating uses of the snippet tag in the JDK API documentation in particular, it is envisaged that we can provide library support within the jtreg framework.

Other kinds of snippet content

Although it is expected that the primary use of the snippet tag will be for Java source code, it is also possible to use snippets for other kinds of content, such as properties files, or plain text such as the output from running a command. The javadoc tool may provide basic highlighting for some of these kinds of additional content.

Generated HTML

The HTML that is generated to present a snippet is deliberately unspecified. However, the generated HTML for each snippet will declare an id such that it can be the target of a link from elsewhere in the documentation. The value for the id will either use the value of the id attribute declared in the snippet tag, if there is such a value available, or a default value will be used.

Alternatives

There are third-party JavaScript solutions to provide syntax highlighting. However, a noteworthy characteristic of JDK API documentation is the desire to provide examples involving new language features, which may not be correctly handled by such solutions in a timely manner. In addition, such solutions are typically based on the use of regular expressions, which can be very fragile, and cannot leverage any additional knowledge that might be available when the documentation is generated.

The use of block comments to specify markup in the snippet content was considered. However, block comments for markup are visually intrusive in the source code, and can only be used in external snippets.

Testing

The feature can be tested using the standard test infrastructure for javadoc features: this includes jtreg tests, and related tools to check for the correctness of the generated documentation.

Risks and Assumptions

It is assumed that there will be a parallel effort to provide tests to validate the code fragments in the existing JDK API documentation. However, that is not a requirement for the success of this feature.

Dependencies

There are no external dependencies.