The Object Teams Blog

Adding team spirit to your objects.

Posts Tagged ‘null

The message is the message

leave a comment »

I have been ranting about bad error messages, so in my own work, error messages better be helpful. At least I try.

As for the recent milestone 6 of the JDT a significant novelty was actually mostly about the wording of a few error/warning messages issued by the null analysis of the JDT compiler. We actually had quite a discussion (no, I’m not expecting you to read all the comments in the bug:)).

Why did we go through all this instead of using the time to fix more bugs? Because the entire business of implementing more compiler warnings and in particular introducing annotation-based null analysis is to help you to better understand possible weaknesses of your code. This means our job isn’t done when the error message is printed on your screen, but only when you recognize why (and possibly how) your code could be improved.

So the game is:
when you see one of the new compiler messages
“what does it tell you?

Intended “inconsistency”

Let’s start with an example that at first looks inconsistent:

Both methods basically have the same code, still lines 14-17 are free of any warnings whereas the corresponding lines 24-27 have one warning and even an error. What does it tell us?

Here are some of the messages:

line 10 Redundant null check: The variable val1 cannot be null at this location
line 12 Null comparison always yields false: The variable val1 cannot be null at this location

Before bug 365859 the second method would show the same messages, giving no clue why after the initial start all the same code gives different results later. The initial improvement in that bug was to update the messages like this:

line 20 Redundant null check: The variable val2 is specified as @NonNull
line 22 Null comparison always yields false: The variable val2 is specified as @NonNull

Alright! Here lies the difference: in the first method, compiler warnings are based on the fact that we see an assignment with the non-null value "OK" and carefully follow each data-flow from there on. Non-null definitely holds until line 15, where potentially (depending on where b takes the control flow) null is assigned. Now the check in line 16 appears useful.

By contrast, the warnings in the second method tell us, that they are not based on flow analysis, but on the mere fact that val2 is declared as of type @NonNull String. This specification is effectual, independent of location and flow, which has two consequences: now the assignment in line 25 is illegal; and since we can’t accept this assignment, line 26 still judges by the declaration of val2 which says: @NonNull:

line 25 Null type mismatch: required ‘@NonNull String’ but the provided value is null
line 26 Redundant null check: The variable val2 is specified as @NonNull

Communicate the reasoning

Three levels to a good error message:

  1. “You did wrong.”
  2. “Your mistake was …”
  3. “This is wrong because …”

By now you probably know what I think of tools that answer using (1). To go from (2) towards (3) we split one particular error message into three. Look at this:

which gives these error messages:

line 31 Null type mismatch: required ‘@NonNull String’ but the provided value is null
line 32 Null type mismatch: required ‘@NonNull String’ but the provided value is specified as @Nullable
line 34 Null type mismatch: required ‘@NonNull String’ but the provided value is inferred as @Nullable

Line 31 is obvious.

Line 32 is wrong because in is declared as @Nullable, saying null is a legal value for in, but since it’s not legal for tmp2 the assignment is wrong.

In line 34 we are assigning a value that has no nullness specification; we say, unknown has a “legacy” type. From that alone the compiler cannot decide whether the assignment in line 34 is good. However, using also information from line 33 we can infer that unknown (probably) has type @Nullable String. In this particular case the inference is obvious, but the steps that lead to such conclusion can be arbitrarily complex.

What does this distinction tell you?

The error in line 31 is a plain violation of the specification: tmp1 is required to be nonnull, but the assigment attempts to definitely break that rule.

The error in line 32 denotes the conflict between two contradictory declarations. We know nothing about actual runtime values, but we can tell without any doubt that the code violates a rule.

Errors of the type in line 34 are reported as a courtesy to the user: you didn’t say what kind of variable unknown is, thus normally the compiler would be reluctant to report problems regarding its use, but looking a bit deeper the compiler can infer some missing information. Only in this category it makes sense to discuss whether the conclusion is correct. The inference inside the compiler might be wrong (which would be a compiler bug).

Sources of uncertainty

Of the previous messages, only the one in line 31 mentions a runtime fact, the remaining errors only refer to possibilities of null values where no null value is allowed. In these cases the program might actually work – by accident. Just like this program might work:

void maybe(Object val) {
   System.out.println(val.toUpperCase());
}

While this is not a legal Java program, a hypothetical compiler could produce runnable byte code, and if the method is invoked with an argument that happens to be a String, all is well – by accident.

While we have no guarantee that things would break at runtime, we know for sure that some rule has been broken and thus the program is rejected.

The following method shows a different kind of uncertainty:

What can we tell about this assignment? Well … we don’t know, it’s not definitely bad, but it’s not good either. What’s the problem? We need a @NonNull value, but we simply have no information whether unspecified can possibly be null or not. One of those legacy types again. After much back and forth we finally found that we have a precendent for this kind of problem: what’s the compiler say to this snippet:

void safety2(List unspecified) {
    List<String> strings = unspecified;
}

Right, it says:

Type safety: The expression of type List needs unchecked conversion to conform to List

meaning: we receive an argument with a type that lacks detailed specification, but we require such details on the left hand side of the assignment. Whether or not the RHS value matches the LHS requirement cannot be checked by the compiler. Argument unspecified has another kind of legacy type: a raw type. To gracefully handle the transition from legacy code to new code with more complete type specifications we only give a warning.

The same for null specifications:

line 41 Null type safety: The expression of type String needs unchecked conversion to conform to ‘@NonNull String’

In both cases, raw types and types lacking null specification, there are situations where ignoring this warning is actually OK: the legacy part of the code may be written in the full intention to conform to the rule (of only putting strings into the list / using only nonnull values), but was not able to express this in the expected style (with type parameters / null annotations). Maybe the information is still documented, e.g., in the javadoc. If you can convince yourself that the code plays by the rules although not declaring to do so: fine. But the compiler cannot check this, so it passes the responsibility to you, along with this warning.

Tuning comiler messages

If you buy into null annotations, the distinction of what is reported as an error vs warning should hopefully be helpful out of the box. Should you wish to change this, please do so with care. Ignoring some errors can render the whole business of null annotations futile. Still we hope that the correspondence between compiler messages and configuration options is clear:

These options directly correspond to the messages shown above:

problems controlled by this option
lines 31,32 Violation of null specification
line 34 Conflict between null annotations and null inference
line 39 Unchecked conversion from non-annotated type to @NonNull type

Conclusion

The compiler does an awful lot of work trying to figure out whether your code makes sense, definitely, or maybe, or maybe not, or definitely not. We just decided, it should try a little harder to also explain its findings. Still, these messages are constrained to be short statements, so another part of the explanation is of course our job: to educate people about the background and rationale why the compiler gives the answers it gives.

I do hope you find the messages helpful, maybe even more so with a little background knowledge.

The next steps will be: what’s a good method for gradually applying null annotations to existing code? And during that process, what’s a good method for reacting to the compiler messages so that from throwing code at the compiler and throwing error messages back we move towards a fruitful dialog, with you as the brilliant Holmes and the compiler your loyal assistant Watson, just a bit quicker than the original, but that’s elementary.

Written by Stephan Herrmann

April 15, 2012 at 01:53

Posted in Eclipse

Tagged with , , , ,

Help the JDT Compiler helping you! – 3: The essence of null annotations

with 2 comments

After my little excursion to detecting resource leaks let me return to our favorite bug: NPE.

Basic support for null annotations was big news at the release of the Eclipse SDK Juno M4.

To fully understand the approach it’s good to sit back and think of what exactly we’re telling the compiler when we add null annotations to a program.

Annotations as filters

One could certainly use the annotations to mean the following:

@NonNull
“I don’t think this variable will ever hold a null, no need to warn me, when I dereference that value.”
@Nullable
“I guess this variable could be null, so please warn me when I forget a null check before dereferencing.”

Interpreted in this way, the annotations would already be helpful, and actually this would rank them in the same category as @SuppressWarnings("null"): no matter what the compiler analyzed up-to this point, please take my word that this thing is / is not dangerous. In both cases we’d ask the compiler to look at certain things and ignore others, because we “know better”.

However, telling the compiler what’s dangerous and what’s not puts the cart before the horse. If I do so, I will be the weakest link in the chain of analysis. I could err, I do err – that’s why I have NPEs in the first place, so I shouldn’t tell the compiler to trust my judgment.

The good news is: when used appropriately null annotations provide a lot more help.

Annotations as an extension of the type system

The essence of static typing

Lets step back and imagine Java wouldn’t have static typing. A variable declaration would be nothing more than a name, introduced -say- using an imaginary keyword var:

   var o;
   o = new Person();
   o.meow();
 

Right, we could now assign any object, any value, to the variable o and on the other hand we could attempt to invoke any method. Only at runtime will we notice whether the object refered to by o actually has a method meow(). Obviously, this code is unsafe, it could abort saying “Message on understood”. As Java programmers we don’t accept this unsafety, so we use types as a specification:

A typed variable declaration adds a specification with these two sides:

  • It adds a constraint such that only certain values can legally be assigned to the variable
  • It establishes a guarantee that certain operations are well-defined wrt the value of the variable.

A statically typed language forces a decision: what values do I want to allow to be bound to the variable? If I declare o as of type Cat the compiler can conclude that “o = new Person();” violates the constraint and cannot be accepted. If, OTOH, I declared o as of type Person the compiler won’t complain at the assignment but typically it will not find a meow() method in a class Person so that line is now illegal. Only if all things match: the declaration, the assignment, and the usage of a variable, only then will the compiler accept the program and certify that this program will not raise “Message not understood”. It’s a trade: constraint for guarantee.
In a statically typed language we are constrained in what we can say, but we gain the guarantee that a certain kind of error will not occur at runtime.

Sounds fair?

More constraints – more guarantees

Standard static typing constrains the program such that values match to the operations we perform with these values, except – there’s a big loop-hole: in traditional object-oriented type systems each class type contains a value that is not suitable for most operations: the value null, which Tony Hoare called his “billion dollar mistake”.

At a closer look, static type checking in Java is founded on a contradiction:

  • When analyzing an assignment assume that null is a legal value for every object type.
  • When looking at a method call / a field reference assume that null cannot occur (otherwise no unchecked dereference could be considered as legal).

This is exactly, what null annotations fix: they split each traditional object type into two types:

@NonNull Cat
This is the type that contains only cat values
@Nullable Cat
This is the union of the above type and the null-type which contains only one value: null

You can read the type @Nullable Cat as: either a cat or null.

Null warnings vs. type errors

Those users who try the new feature in the JDT may be surprised to see a whole new kind of error messages. While the original goal is to get alerted about potential NPEs, the compiler may now complain with messages like:

Type mismatch: required ‘@NonNull Cat’ but the provided value can be null

The question may arise, why this is reported as an error, even if no NPE can be directly caused at the statement in question. The answer can be deduced from the following analogy:

void foo(Object o, @Nullable Cat c) {
    Cat aCat = o;                 // "Type mismatch: cannot convert from Object to Cat"
    @NonNull Cat reallyACat  = c; // "Type mismatch: required '@NonNull Cat' but the provided value can be null."
}
 

(The wording of the second message will be still improved to better reflect different kinds of RHS values).

The analogy shows:

  • The assignment itself could actually succeed, and even if types don’t match, a language without static typing could actually accept both assignments.
  • If, however, the assignment were accepted, all subsequent analysis of the use of this variable is useless, because the assumption about the variable’s type may be broken.

Therefor, a first step towards making NPE impossible is to be strict about these rules. Assigning a value to a @NonNull variable without being able to prove that the value is not null is illegal. Just as assigning an Object value to a Cat variable without being able to prove that the value is indeed a cat is illegal.

Interestingly, for the first assignment, Java offers a workaround:

    Cat aCat = (Cat) o;
 

Using the cast operator has two implications: we tell the compiler that we “know better”, that o is actually a cat (we do believe so) and secondly, as compiler and JVM cannot fully trust our judgment a check operation will be generated that will raise a ClassCastException if our assumption was wrong.

Can we do something similar for @NonNull conversion? Without the help of JSR 308 we cannot use annotations in a cast, but we can use a little helper:

void foo(Object o, @Nullable Cat c) {
    @NonNull Cat reallyACat  = assertNonNull(c);
}
@NonNull  T assertNonNull(T val) {
    if (val == null) throw new NullPointerException("NonNull assertion violated");
    return val;
}
 

corrected on 2012/03/05

What? We deliberately throw an NPE although the value isn’t even dereferenced? Why that?

The helper mimics exactly what a cast does for normal type conversions: check if the given value conforms to the required type. If not, raise an exception. If the check succeeds re-type the value to the required type.

Here’s an old school alternative:

void foo(@Nullable Cat c) {
    @SuppressWarnings("null") @NonNull Cat reallyACat  = c;
}
 

(Requires that you enable using @SuppressWarnings for optional errors).

Which approach is better? Throwing an exception as soon as something unexpected happens is far better than silencing the warning and waiting for it to explode sometime later at some other location in the code. The difference is felt during debugging. It’s about blame assignment.
If things blow up at runtime, I want to know which part of the code caused the problem. If I use @SuppressWarnings that part is in stealth mode, and an innocent part of the code will get the blame when it uses the wrong-typed value.

Remember, however, that cast and assertNonNull are not the solution, those are workarounds. Solutions must explicitly perform the check and provide application specific behavior to both outcomes of the check. Just as a cast without an instanceof check is still a land-mine, so is the use of the above helper: NPE can still occur. If you need to dereference a variable that’s not @NonNull you should really ask yourself:

  • How can it happen that I end up with a null value in this position?
  • How can the application safely and soundly continue in that situation?

These questions cannot be answered by any tool, these relate to the design of your software.

Help the JDT compiler helping you

This post showed you two things you can and should do to help the compiler helping you:

Add null annotations to resolve the contradiction that’s inherent in Java’s type system: a type can only either contain the value null or not contain the value null. Still Java’s type system opportunistically assumes a little bit of both. With annotations you can resolve the ambiguity and state which of the two possible types you mean.

Second, listen to the new type error messages. They’re fundamental to the analysis. If you disregard (or even disable) these messages there’s no point in letting the analysis apply all its sophisticated machinery. From false assumptions we cannot conclude anything useful.

If you apply these two hints, the compiler will be your friend and report quite some interesting findings. For a project that uses null annotations right from the first line of code written, this advice should be enough. The difficult part is: if you have a large existing code base already, the compiler will have a lot to complain. Think of migrating a fully untyped program to Java. You bet you could use some more help here. Let’s talk about that in future posts.

Written by Stephan Herrmann

February 21, 2012 at 20:39

Posted in Eclipse

Tagged with , , , ,

Object Teams with Null Annotations

with 5 comments

The recent release of Juno M4 brought an interesting combination: The Object Teams Development Tooling now natively supports annotation-based null analysis for Object Teams (OT/J). How about that? 🙂
NO NPE

The path behind us

Annotation-based null analysis has been added to Eclipse in several stages:

Using OT/J for prototyping
As discussed in this post, OT/J excelled once more in a complex development challenge: it solved the conflict between extremely tight integration and separate development without double maintenance. That part was real fun.
Applying the prototype to numerous platforms
Next I reported that only one binary deployment of the OT/J-based prototype sufficed to upgrade any of 12 different versions of the JDT to support null annotations — looks like a cool product line
Pushing the prototype into the JDT/Core
Next all of the JDT team (Core and UI) invested efforts to make the new feature an integral part of the JDT. Thanks to all for this great collaboration!
Merging the changes into the OTDT
Now, that the new stuff was mixed back into the plain-Java implementation of the JDT, it was no longer applicable to other variants, but the routine merge between JDT/Core HEAD and Object Teams automatically brought it back for us. With the OTDT 2.1 M4, annotation-based null analysis is integral part of the OTDT.

Where we are now

Regarding the JDT, others like Andrey, Deepak and Aysush have beaten me in blogging about the new coolness. It seems the feature even made it to become a top mention of the Eclipse SDK Juno M4. Thanks for spreading the word!

Ah, and thanks to FOSSLC you can now watch my ECE 2011 presentation on this topic.

Two problems of OOP, and their solutions

Now, OT/J with null annotations is indeed an interesting mix, because it solves two inherent problems of object-oriented programming, which couldn’t differ more:

1.: NullPointerException is the most widespread and most embarrassing bug that we produce day after day, again and again. Pushing support for null annotations into the JDT has one major motivation: if you use the JDT but don’t use null annotations you’ll no longer have an excuse. For no good reasons your code will retain these miserable properties:

  • It will throw those embarrassing NPEs.
  • It doesn’t tell the reader about fundamental design decisions: which part of the code is responsible for handling which potential problems?

Why is this problem inherent to OOP? The dangerous operator that causes the exception is this:

right, the tiny little dot. And that happens to be the least dispensable operator in OOP.

2.: Objectivity seems to be a central property on any approach that is based just on Objects. While so many other activities in software engineering are based on the insight that complex problems with many stakeholders involved can best be addressed using perspectives and views etc., OOP forces you to abandon all that: an object is an object is an object. Think of a very simple object: a File. Some part of the application will be interested in the content so it can decode the bytes and do s.t. meaningful with it, another part of the application (maybe an underlying framework) will mainly be interested in the path in the filesystem and how it can be protected against concurrent writing, still other parts don’t care about either but only let you send the thing over the net. By representing the “File” as an object, that object must have all properties that are relevant to any part of the application. It must be openable, lockable and sendable and whatnot. This yields
bloated objects and unnecessary, sometimes daunting dependencies. Inside the object all those different use cases it is involved in can not be separated!

With roles objectivity is replaced by a disciplined form of subjectivity: each part of the application will see the object with exactly those properties it needs, mediated by a specific role. New parts can add new properties to existing objects — but not in the unsafe style of dynamic languages, but strictly typed and checked. What does it mean for practical design challenges? E.g, direct support for feature oriented designs – the direct path to painless product lines etc.

Just like the dot, objectivity seems to be hardcoded into OOP. While null annotations make the dot safe(r), the roles and teams of OT/J add a new dimension to OOP where perspectives can be used directly in the implementation. Maybe it does make sense, to have both capabilities in one language 🙂 although one of them cleans up what should have been sorted out many decades ago while the other opens new doors towards the future of sustainable software designs.

The road ahead

The work on null annotations goes on. What we have in M4 is usable and I can only encourage adopters to start using it right now, but we still have an ambitious goal: eventually, the null analysis shall not only find some NPEs in your program, but eventually the absense of null related errors and warnings shall give the developer the guarantee that this piece of code will never throw NPE at runtime.

What’s missing towards that goal:

  1. Fields: we don’t yet support null annotations for fields. This is next on our plan, but one particular issue will require experimentation and feedback: how do we handle the initialization phase of an object, where fields start as being null? More on that soon.
  2. Libraries: we want to support null specifications for libraries that have no null annotations in their source code.
  3. JSR 308: only with JSR 308 will we be able to annotate all occurrences of types, like, e.g., the element type of a collection (think of List)

Please stay tuned as the feature evolves. Feedback including bug reports is very welcome!

Ah, and one more thing in the future: I finally have the opportunity to work out a cool tutorial with a fellow JDT committer: How To Train the JDT Dragon with Ayushman. Hope to see y’all in Reston!

Written by Stephan Herrmann

December 20, 2011 at 22:32

Mix-n-match language support

leave a comment »

I’ve been involved in the release of different versions of the JDT lately, supporting different flavors of Java.

Classical release management

At the core we have the plain JDT, of which we published the 3.7.0 release in June and right now first release candidates are being prepared towards the 3.7.1 service release, which will be the first official release to support Java 7. At the same time the first milestones towards 3.8 are being built. OK, this is almost normal business — with the exception of the service release differs more than usual from its base release, due to the unhappy timing of the release of Java 7 vs. Eclipse 3.7.

So that’s 3 versions in 2 month’s time.

First variant: Object Teams

The same release plan is mirrored by the Object Teams releases 2.0.0, 2.0.1RC1, 2.1.0M1. Merging the delta from JDT 3.7 to 3.7.1 into the OTDT was a challenge, given that this delta contained the full implementation of all that’s new in Java 7. Still with the experience of regularly merging JDT/Core changes into the OT variant, the pure merging was less than one day plus a couple more days until all 50000+ tests were green again. The nice thing about the architecture of the OTDT: after merging the JDT/Core, I was done. Since all other adaptations of the JDT are implemented using OT/Equinox adopting, e.g., all the new quick assists for Java 7 required a total of zero minutes integration time.

I took the liberty of branching 2.0.x and 2.1 only after integrating the Java 7 support, which also means that 2.1 M1 has only a small number of OT-specific improvements that did not already go into 2.0.1.

This gives 6 versions of the JDT in 2 month’s time.

Prototyping annotation based null analysis

As I wrote before, I’m preparing a comprehensive new feature for the JDT/Core: static analysis for potential NullPointerException based on annotations in the code. The latest patch attached to the bug had almost 3000 lines. Recent discussions at ECOOP made me change my mind in a few questions, so I changed some implementation strategies. Luckily the code is well modularized due to the use of OT/Equinox.

Now came the big question: against which version of the JDT should I build the null-annotation add-on? I mean, which of the 6 versions I have been involved in during the last 2 months?

As I like a fair challenge every now and then I decided: all six, i.e., I wanted to support adding the new static analysis to all six JDT versions mentioned before.

Integration details

Anybody who has worked on a Java compiler will confirm: if you change one feature of the compiler chances are that any other feature can be broken by the change (I’m just paraphrasing: “it’s complex”). And indeed, applying the nullity plug-in to the OTDT caused some headache at first, because both variants of the compiler make specific assumptions about the order in which specific information is available during the compilation process. It turned out that two of these assumptions where simply incompatible, so I had to make some changes (here I made the null analysis more robust).

At the point where I thought I was done, I tripped over an ugly problem that’s intrinsic to Java.
The nullity plug-in adapts a method in the JDT/Core which contains the following switch statement:

        while ((token = this.scanner.getNextToken()) != TerminalTokens.TokenNameEOF) {
                IExtendedModifier modifier = null;
                switch(token) {
                        case TerminalTokens.TokenNameabstract:
                                modifier = createModifier(Modifier.ModifierKeyword.ABSTRACT_KEYWORD);
                                break;
                        case TerminalTokens.TokenNamepublic:
                                modifier = createModifier(Modifier.ModifierKeyword.PUBLIC_KEYWORD);
                                break;
                        // more cases
                }
        }
 

I have a copy of this method where I only added a few lines to one of the case blocks.
Compiles fine against any version of the JDT. But Eclipse hangs when I install this plugin on top of a wrong JDT version. What’s wrong?

The problem lies in the (internal) interface TerminalTokens. The required constants TokenNameabstract etc. are of course present in all versions of this interface, however the values of these constants change every time the parser is generated anew. If constants were really abstractions that encapsulate their implementation values, all would be fine, but the Java byte code knows nothing about such an abstraction, all constant values are inlined during compilation. In other words: the meaning of a constant depends solely on the definitions which the compiler sees during compilation. Thus compiling the above switch statement hardcodes a dependency on one particular version of the interface TerminalTokens. BAD.

After recognizing the problem, I had to copy some different versions of the interface into my plug-in, implement some logic to translate between the different encodings and that problem was solved.

What’s next?

Nothing is next. At this point I could apply the nullity plug-in to all six versions of the JDT and all are behaving well.

We indeed have 12 versions of the JDT in 2 month’s time.

Mix-n-match

Would you like Java with our without the version 7 enhancements (stable release or milestone)? May I add some role and team classes? How about a dash more static analysis? It turns out we have more than just one product, we have a full little product line with features to pick or opt-out:

 

Java 6

Java 7
Indigo

Indigo SR1

Juno M1
no null annotations

Plain JDT

 

 

 
OTDT

 

 

 
with null annotations

Plain JDT

 

 

 
OTDT

 

 

 

Just make your choice 🙂
Happy hacking with null annotations and try-with-resources in OT/J.

EclipseCon Europe 2011  

BTW: if you want to hear a bit more about the work on null annotations, you should really come to EclipseCon Europe — why not drop a comment at this submission 🙂

Written by Stephan Herrmann

August 19, 2011 at 17:22