JEP draft: Add detailed message to NullPointerException describing what is null

AuthorsGoetz Lindenmaier, Ralf Schmelter
OwnerGoetz Lindenmaier
TypeFeature
ScopeJDK
StatusDraft
Componenthotspot / runtime
Discussionhotspot dash runtime dash dev at openjdk dot java dot net, core dash lib dash dev at openjdk dot java dot net
EffortS
DurationS
Created2019/03/15 10:27
Updated2019/03/29 14:03
Issue8220715

Summary

NullPointerExceptions are freqently encountered developing or maintaining a Java application. NullPointerExceptions often don't contain a message. This complicates finding the cause of the exception.

This JEP proposes to enhance the exception text to tell what was null and which action failed.

Motivation

Currently, a common NullPointerException does not contain a helpful message. The following code:

a.to_b.to_c = null;
a.to_b.to_c.to_d.num = 99;

will just print:

java.lang.NullPointerException

This does not help to find out what was null. Was it a? a.to_b? a.to_b.to_c?

A message like

'a.to_b.to_c' is null. Can not read field 'to_d'.

will immediately tell where the exception was thrown. For more message examples see [1].

The same algorithm can be used to improve other messages:

a[i][j][k]; // Index j is out-of-bounds

currently prints

java.lang.ArrayIndexOutOfBoundsException: Index 99 out of bounds for length 2

This message does not tell which index is out of bounds. This can be improved to print:

java.lang.ArrayIndexOutOfBoundsException: a[i][j]: Index j = 99 out of bounds for length 2

Description

Basic algorithm to compute the message

Java code like a.to_b.to_c is compiled to several bytecode instructions. When the exception is raised, the original Java code is not available, but the bytecodes. Given the bytecodes of the method in which the exception was raised and the precise bytecode index where it happened, information about the exeption context is derived and printed.

The message consists of two parts.

The first part tells what code resulted in a null value.

The second part tells the action that can not be performed on the null value.

The second part can be computed straight forward. Looking at the bytecode at the given bytecode index one can just print the message. E.g., for a getfield bytecode, "can not read field <fieldname>" can be printed.

The first part is more complex to be printed. The algorithm walks back to the bytecode that pushed the value popped by the current bytecode on the operand stack. For the getfield example, this is the bytecode that pushed the Object reference that is null on the operand stack. Then a message based on this bytecode is printed.

If this happens to be another getfield bytecode, ".<fieldname> is null" is printed.
Walking back further, a comprehensive text as "a.to_b.to_c is null" is assembled.

Given the bytecode, it is not obvious which previous instruction pushed the null value. To find out about this, a simple data flow analysis is run on the bytecodes.

This data flow analysis walks the bytecodes forward simulating the operand stack. The simulated stack does not contain the values computed by the bytecodes, but information about which bytecode pushed the value to the stack. Therefore this analysis terminates quickly. The analysis runs until the information for the bytecode that raised the exception is available. Thus, it stops before analysing the whole method [3].

To start out with printing the message, the bytecodes and the byte code index are needed. This information happens to be stored in the backtrace datastructure contained in a Throwable. This is an intermediate data structure used to lazily compute the StackTraceElement[] returned by Throwable.getStackTrace().

Message content

The proposed generated message is only printed if the NullPointerException is raised by the runtime. If the exception is explicitly constructed, it makes no sense to add the message. It will be misleading, as no real null value was encountered at the byte code index where the exception is constructed.

The message can not regain the original code 1:1. The message should try though to give as much resemblance to code as possible. This makes it easy to understand and compact.

The message mentions information from the code like class names, method names, field names and variable names. These are printed enclosed in single quotes.

Method signatures are printed in Java syntax not using the JNI signatures.

Local variable names are retrieved from the debug information. If a class file does not contain this information, a replacement must be found. We print <local1>, <local2>, <parameter1> etc.

No message is printed if the method is not listed in the stack trace of the exception. This is the case for frames of hidden methods.

The first part of the message must be implemented for bytecodes that can push object references onto the operand stack, and for those bytecodes that push values that are used recursively (like the array index). This table lists what is printed for which bytecodes. The message is assembled by recursively walking the bytecodes backwards. To limit the output, the recursion depth is limited to 5. This value is arbitrarily chosen.

| bytecode | message printed | | --- | --- | | aconst_null | "'null'" | | array load | If the maximum recursion depth is reached, print "<array>", else recur to the bytecode that pushed the array reference. Print "[". Recur to the bytecode that pushed the index. If this did not print anything, print "...". Print "]". | | const, bipush, sipush | Print the constant, suffixed with "f" of "L" if appropriate. | | getfield | Recur to the bytecode that pushed the reference that is accessed here onto the operand stack. If this did print something, print ".". Print the field name. | | getstatic | Print "static <classname>.<fieldname>" | | invokeinterface, invokevirtual, invokespecial, invokestatic | If in the first recursion, print "The return value of '<methodname>'". Else only print the method name. Don't recur for the sender. We do not print actual parameters. | | load | If a local variable table is available, print the variable name. Else, print "this" or "<parameter n >" or "<local n >" for given n. |

This message is appended with " is null. "

This yields strings as "'static A.b' is null. ", "'a[j][1][...][k]' is null. ", "'<array>[0][0][0][0][0]' is null. ", "The return value of 'A.b()' is null. " etc.

Handling of other bytecodes can be added, but seemed not necessary. Only complex index expressions in arrays will be omitted, "..." is printed instead. dub bytecodes are handled by the dataflow analysis.

For the second part of the message, bytecodes that can raise a null pointer exception, i.e., bytecodes the dereference a reference placed on the operand stack, must be handled. The following messages are used:

| bytecode | message printed | | --- | --- | | aload | "Can not load from null <element type> array." | | arraylength | "Can not read the array length." | | astore | "Can not store to null <element type> array." | | athrow | "Can not throw a null exception object." | | getfield | "Can not read field '<fieldname>'." | | invokeinterface, invokespecial, invokevirtual |"Can not invoke method '<methodname>'." | | monitorenter | "Can not enter a null monitor." | | monitorexit | "Can not exit a null monitor." | | putfield | "Can not write field '<fieldname>'." |

<classname>, <fieldname>, <methodname> and <element type> are replaced with the respective strings. If other bytecodes are encountered, no message is printed.

Different bytecode variants of a method

Given a class name and a method name, the virtual machine can use several different variants of a method. Different classes with the same name containing methods with the same name can be loaded by different class loaders.

Due to redefinition and bytecode modification different variants of the same method can be used by the jvm implementation. The bytecode used can differ from the bytecodes in the original class file that was loaded. The analysis must operate on the very bytecode that has been executed when the exception was raised. The bytecode index stored in the exception must point to the instruction that caused the exception.

Using the class, method index and bytecode index attached to the backtrace, this is guaranteed by construction. The method representation referenced from the backtrace is always the right one.

Computation overhead

When a NullPointerException is thrown, a NullPointerException object is created. Usually, exception messages are computed at Object creation time and stored in the private field 'detailMessage' of Throwable. As computing the NullPointerException message proposed here is a considerable overhead, this would slow down throwing NullPointerExceptions.

Fortunately, most Exceptions are discarded without looking at the message. I.e, if we delay computing the message until it is actually accessed, we omit the effort for all the exceptions that are discarded without printing the message.

To compute the message delayed, access to the method containing the bytecode and the bytecode index must be preserved until the message is computed. The backtrace datastructure is preserved for a long time anyways, so this is given by construction.

Hidden frames are not preserved in the backtrace datastructure. Looking at the backtrace one can not tell whether the 'real' top frame was dropped. Therefore the backtrace is extended to store information whether this happened.[6]

Message persistance

Usually, an exception message is passed to the constructor of Throwable that writes it to its private field 'detailMessage'. If we compute the message only on access, we can no more pass it to the Throwable constuctor. As the field is private, there is no natural way to store the message in it. As a consequence, the message is not persisted but recomputed on every access. Also, npe.getMessage() == npe.getMessage() does not hold, as it is the case for other exceptions.

If a Throwable is serialized, the backtrace is gone. If the backtrace is lost, the message can no more be computed. I.e., getMessage() called on a serialized NPE will return null.

Cases not covered by this JEP

If the proper method and bytecodes can not be reached from the backtrace data structure, no message will be printed.

If the 'real' top frame is a hidden frame, and -XX:-ShowHiddenFrames is set (which is the default), no message is printed. It would be misleading to report a method from a frame that is not visible in the stack trace.

If the class file does not contain debug information, field and parameter names are not printed. Instead placeholders like <local1>, <local2>, <parameter1> etc. are printed.

If the NullPointerException was serialized, no message is printed.

Alternatives

Making the message persistant.

Current discussion lead to the decision not to persist the message, but to recompute it whenever Throwable.getMessage() is called.

Several ways exists to make the message persistant, though.

Throwable is not immutable in first place. StackTraceElement[] is computed after construction.

Similar, the message can be written to Throwable.detailMessage in the first call to getMessage().

There are several possibilities to implement this:

To persist the message in serialization writeReplace() can be implemented.

Implementing the JEP in Java

The current proposal is to implement this in the Java runtime in C++ accessing directly the available datastructures in the metaspace.

Alternatively, this can be implemented in Java.

A Java implementation also needs the bytecodes of the method that raised the exception and the byte code index.

The byte code index is not available in StackTraceElement[0], which only contains Strings of the class and method names. StackWalker's StackFrame can be used to obtain the needed information from the top frame. It gives access to the proper Class containing the method and the byte code index.

ASM can be used to implement the data flow analysis and printing the messages based on the bytecodes.

The current implementation of StackWalker requires that is is obtained in the NullPointerException constructor. This is not optimal wrt. performance as detailed above. It would be possible to extend StackWalker to operate on Throwables using the internal backtrace data structure to generate it lazily.

ASM does not have a notion of bytecode indexes. I.e., given a bytecode index, you can not access the corresponding bytecode. It is straight forward to extend ASM to compute this information when parsing the bytecodes. Complex bytecodes as tableswitch must be handled.

ASM should operate on the actual bytecode used. A file
containing the class might not be available or hard to locate. The bytecode run by the virtual machine may be modified in many ways. It might be possible to use the information returned by JvmtiClassFileReconstitutor.

Testing

Major testing is done by the unit test as it requests the NullPointerException message excercising the new code.

To avoid regressions some larger amounts of code should be run. The jtreg tests should be run to detect other tests that handle the message and need to be adapted.

There are no special hardware or platform requirements.

Risks and Assumptions

This imposes overhead on retrieving the message of a NullPointerMessage.

The risk of breaking something in the virtual machine is low. The feature can be implemented completely on top of existing data structures without changeing them. Only writing the 'detailMessage' field of Throwable late is a considerable change.

The implementation needs to be extended if new bytecodes are added to the Bytecode specification.

The concern was raised that printing field names or local names might impose a security problem. If so, this should be addressed more globally. Other exceptions already print field information (IllegalAccessError). Class and method names are contained in the stack trace. Printing field names in NullPointerExceptions should not impose any new security problems.

Dependencies

None. A stable and tested prototype exists.

References

A prototype has been proposed in 8218628: Add detailed message to NullPointerException describing what is null..

A webrev of the prototype is available.

[1] Messages if classfiles contain debug info. Messages if classfiles contain no debug info.

[2] java_lang_Throwable::get_method_and_bci(...) retrieves the needed information of the top frame.

For the following references see prims.cpp: JVM_GetExtendedNPEMessage(...) in the webrev:

[3] The data flow analysis of the bytecodes is implemented in the TrackingStackCreator constructor.

[4] The message part for the failed action is computed by TrackingStackCreator.get_null_pointer_slot(...).

[5] The message part describing the null entity is computed by TrackingStackCreator.get_source(...).

[6] A solution is proposed in 8221077: No NullPointerException message if top frame is hidden.