The Hitchhiker's Guide to javac

DON'T PANIC

The following notes are provided to help newcomers to javac navigate their way around the code base.

See also:

Functional blocks

javac can be considered as a number of functional blocks: some go towards the abstraction of a Java compiler, that can transform Java source files into Java class files, while others provide the environmental infrastructure to support the compiler.

area block description
environment

provides compiler support

Java Compiler API (JSR 199)
invocation how to invoke javac
options option handling
files file manager
reporting log, diagnostics, etc
utils utility classes
Java compiler

transforms source files into class files

JLS, JVMS, Language Model API, Annotation Processing API (JSR 269), Compiler Tree API
syntax tree AST, parser
semantics semantic analysis of AST
processing annotation processing
simplification simplify the AST
code generation generating class files

Environment

Invocation

package classes
javax.tools ToolProvider, JavaCompiler
com.sun.source.util JavacTask
com.sun.tools.javac Main
com.sun.tools.javac.api JavacTool
com.sun.tools.javac.main Main, JavaCompiler

These are the classes for running javac. You can invoke javac in three ways:

Main
Simple, direct command line functionality.
Java Compiler API (JSR 199)
Access via classes in the javax.tools package.
Compiler Tree API
Cast a javax.tools.CompilationTask to a com.sun.source.util.JavacTask.

All the invocation paths go through com.sun.tools.javac.main.JavaCompiler, which is the main compiler driver.

Option Handling

package classes
com.sun.tools.javac.main OptionName, JavacOption, RecognizedOptions
com.sun.tools.javac.api JavacTool
com.sun.source.util Options

Currently, there are different option decoding paths for Main and for the Compiler/Tree API. (This is not good). This is being rationalized, with OptionName, JavacOption and RecognizedOptions being combined into a single new enum Option.

FileManager

package classes
javax.tools JavaFileManager, FileObject, JavaFileObject
com.sun.tools.javac.file JavacFileManager, Paths
com.sun.tools.javac.nio JavacPathFileManager

JavaFileManager provides a very simple abstract file system, sufficient for the needs of javac.

The implementation of JavaFileManager in the javac.file package uses java.io.File and java.util.jar API, and is currently the standard (default) file manager.

The implementation of JavaFileManager in the javac.nio package uses java.nio.file.Path and is recommended for future use by anyone wanting to have javac access files via an abstract file system.

Reporting

package classes
javax.tools Diagnostic, DiagnosticListener
com.sun.tools.javac.util AbstractLog, Log, DiagnosticFormatter and subtypes, JCDiagnostic

The Log provides the primary mechanism for reporting diagnostics, and these days provides a flexible reporting mechanism, based on configurable, pluggable formatters.

To facilitate testing, there is a raw diagnostics mode which can print out the basic details of a diagnostic, in a form which is safe for use in golden files.

For more information, see javac diagnostics.

Utils

package classes
com.sun.tools.javac.util Context, List, Name, NameTable, ...

The root of all information for a compilation is a Context. Originally, it was one per compilation; currently, if annotation processing is involved, it is one context per annotation processing round and one for the final compilation, although an increasing amount of information is migrated between contexts. It is a goal to get back to one context per compilation.

javac uses its own implementation of List, which provides simple, fast, immutable lists.

Name and NameTable provide simple interned strings.

Java Compiler

Syntax Tree

package classes
com.sun.source.tree Tree and subtypes
com.sun.tools.javac.tree JCTree and subtypes, TreeInfo
com.sun.tools.javac.parser Scanner, JavacParser, etc

The Scanner class maps source text into a stream of tokens. The JavacParser class converts the stream of tokens into one or more syntax trees.

The classes in the com.sun.source.tree package provide public read access to the syntax trees.

The trees have a “minimalist” API design. Expressions and declarations point to semantic information derived from the tree. Use utility classes and visitors to process trees.

Semantics – The data model

package classes
javax.lang.model.element Element and subtypes
javax.lang.model.type TypeMirror and subtypes
javax.lang.model.util Elements, Types, etc.
com.sun.tools.javac.code Symbol, Type, Annotations, Attribute, TypeAttribute, Types

A Symbol provides semantic information about a declaration. Each subtype of Symbol implements a corresponding subtype of Element.

A Type provides semantic information about an expression. Each subtype of Type implements a corresponding subtype of TypeMirror.

An Attribute provides semantic information about an annotation. Note: They should not be confused with class file attributes, which are completely different.

The Types class provides utility methods for types and symbols.

Semantics – Creating symbol tables

package classes
com.sun.tools.javac.comp Enter, MemberEnter, Annotate

Enter and MemberEnter scan trees for class and member declarations, and initialize symbols.

Annotate scans trees for annotations on class and member declarations, to add information to the symbols prior to annotation processing.

Semantics – Code analysis

package classes
com.sun.tools.javac.comp Attr, Check, Infer, Resolve, Flow

Attr, Check, Infer and Resolve analyze all names and expressions within the program.

Flow performs static program flow analysis. It checks the following:

Annotation Processing

package classes
javax.annotation.processing AbstractProcessor, ProcessingEnvironment, RoundEnvironment
com.sun.tools.javac.model JavacElements, JavacTypes, etc
com.sun.tools.javac.processing JavacProcessingEnvironment

Annotation processing occurs after symbols have been entered (Enter, MemberEnter) but before method analysis (Attr, Flow).

Annotation processing gives users the opportunity to execute code that can analyze method signatures and generate additional types to be used in the compilation.

If new files are generated by the user code, the compilation is conceptually restarted.

Code Simplification

package classes
com.sun.tools.javac.comp Lower, TransTypes
com.sun.tools.javac.tree TreeTranslator

Lower converts “syntactic sugar” constructions into simpler code. This includes inner classes, class literals, assertions, foreach loops, strings in switch, etc., which can all be represented by code not using those constructs.

TransTypes eliminates (erases) generics from the program.

Lower and TransTypes use TreeTranslator to rewrite trees.

Code Generation

package classes
com.sun.tools.javac.jvm all

The Gen and Code classes generate bytecodes for the class file Code attribute.

The Pool class is used to model the class file constant pool.

The CRTable and CRTable classes are used for mapping source positions to bytecodes.

Class files are written out, via the file manager, using ClassWriter.

Source code

Master repository for JDK 8
http://hg.openjdk.java.net/jdk8/jdk8/langtools
Integration repository for JDK 8
http://hg.openjdk.java.net/jdk8/tl/langtools

Testing

Developer tests are in the langtools repository, test directory. Tests are normally run with the jtreg test harness, although all tests can be run either stand-alone or using TestNG (depending on the nature of the test.)

All the tests should pass, all the time, on all platforms.

See Writing jtreg tests and Guidelines for tests in the “langtools” repository

TCK tests are licensed separately.

For more info …