The Object Teams Blog

Adding team spirit to your objects.

Posts Tagged ‘analysis

Help the JDT Compiler helping you! – 3: The essence of null annotations

with 2 comments

After my little excursion to detecting resource leaks let me return to our favorite bug: NPE.

Basic support for null annotations was big news at the release of the Eclipse SDK Juno M4.

To fully understand the approach it’s good to sit back and think of what exactly we’re telling the compiler when we add null annotations to a program.

Annotations as filters

One could certainly use the annotations to mean the following:

@NonNull
“I don’t think this variable will ever hold a null, no need to warn me, when I dereference that value.”
@Nullable
“I guess this variable could be null, so please warn me when I forget a null check before dereferencing.”

Interpreted in this way, the annotations would already be helpful, and actually this would rank them in the same category as @SuppressWarnings("null"): no matter what the compiler analyzed up-to this point, please take my word that this thing is / is not dangerous. In both cases we’d ask the compiler to look at certain things and ignore others, because we “know better”.

However, telling the compiler what’s dangerous and what’s not puts the cart before the horse. If I do so, I will be the weakest link in the chain of analysis. I could err, I do err – that’s why I have NPEs in the first place, so I shouldn’t tell the compiler to trust my judgment.

The good news is: when used appropriately null annotations provide a lot more help.

Annotations as an extension of the type system

The essence of static typing

Lets step back and imagine Java wouldn’t have static typing. A variable declaration would be nothing more than a name, introduced -say- using an imaginary keyword var:

   var o;
   o = new Person();
   o.meow();
 

Right, we could now assign any object, any value, to the variable o and on the other hand we could attempt to invoke any method. Only at runtime will we notice whether the object refered to by o actually has a method meow(). Obviously, this code is unsafe, it could abort saying “Message on understood”. As Java programmers we don’t accept this unsafety, so we use types as a specification:

A typed variable declaration adds a specification with these two sides:

  • It adds a constraint such that only certain values can legally be assigned to the variable
  • It establishes a guarantee that certain operations are well-defined wrt the value of the variable.

A statically typed language forces a decision: what values do I want to allow to be bound to the variable? If I declare o as of type Cat the compiler can conclude that “o = new Person();” violates the constraint and cannot be accepted. If, OTOH, I declared o as of type Person the compiler won’t complain at the assignment but typically it will not find a meow() method in a class Person so that line is now illegal. Only if all things match: the declaration, the assignment, and the usage of a variable, only then will the compiler accept the program and certify that this program will not raise “Message not understood”. It’s a trade: constraint for guarantee.
In a statically typed language we are constrained in what we can say, but we gain the guarantee that a certain kind of error will not occur at runtime.

Sounds fair?

More constraints – more guarantees

Standard static typing constrains the program such that values match to the operations we perform with these values, except – there’s a big loop-hole: in traditional object-oriented type systems each class type contains a value that is not suitable for most operations: the value null, which Tony Hoare called his “billion dollar mistake”.

At a closer look, static type checking in Java is founded on a contradiction:

  • When analyzing an assignment assume that null is a legal value for every object type.
  • When looking at a method call / a field reference assume that null cannot occur (otherwise no unchecked dereference could be considered as legal).

This is exactly, what null annotations fix: they split each traditional object type into two types:

@NonNull Cat
This is the type that contains only cat values
@Nullable Cat
This is the union of the above type and the null-type which contains only one value: null

You can read the type @Nullable Cat as: either a cat or null.

Null warnings vs. type errors

Those users who try the new feature in the JDT may be surprised to see a whole new kind of error messages. While the original goal is to get alerted about potential NPEs, the compiler may now complain with messages like:

Type mismatch: required ‘@NonNull Cat’ but the provided value can be null

The question may arise, why this is reported as an error, even if no NPE can be directly caused at the statement in question. The answer can be deduced from the following analogy:

void foo(Object o, @Nullable Cat c) {
    Cat aCat = o;                 // "Type mismatch: cannot convert from Object to Cat"
    @NonNull Cat reallyACat  = c; // "Type mismatch: required '@NonNull Cat' but the provided value can be null."
}
 

(The wording of the second message will be still improved to better reflect different kinds of RHS values).

The analogy shows:

  • The assignment itself could actually succeed, and even if types don’t match, a language without static typing could actually accept both assignments.
  • If, however, the assignment were accepted, all subsequent analysis of the use of this variable is useless, because the assumption about the variable’s type may be broken.

Therefor, a first step towards making NPE impossible is to be strict about these rules. Assigning a value to a @NonNull variable without being able to prove that the value is not null is illegal. Just as assigning an Object value to a Cat variable without being able to prove that the value is indeed a cat is illegal.

Interestingly, for the first assignment, Java offers a workaround:

    Cat aCat = (Cat) o;
 

Using the cast operator has two implications: we tell the compiler that we “know better”, that o is actually a cat (we do believe so) and secondly, as compiler and JVM cannot fully trust our judgment a check operation will be generated that will raise a ClassCastException if our assumption was wrong.

Can we do something similar for @NonNull conversion? Without the help of JSR 308 we cannot use annotations in a cast, but we can use a little helper:

void foo(Object o, @Nullable Cat c) {
    @NonNull Cat reallyACat  = assertNonNull(c);
}
@NonNull  T assertNonNull(T val) {
    if (val == null) throw new NullPointerException("NonNull assertion violated");
    return val;
}
 

corrected on 2012/03/05

What? We deliberately throw an NPE although the value isn’t even dereferenced? Why that?

The helper mimics exactly what a cast does for normal type conversions: check if the given value conforms to the required type. If not, raise an exception. If the check succeeds re-type the value to the required type.

Here’s an old school alternative:

void foo(@Nullable Cat c) {
    @SuppressWarnings("null") @NonNull Cat reallyACat  = c;
}
 

(Requires that you enable using @SuppressWarnings for optional errors).

Which approach is better? Throwing an exception as soon as something unexpected happens is far better than silencing the warning and waiting for it to explode sometime later at some other location in the code. The difference is felt during debugging. It’s about blame assignment.
If things blow up at runtime, I want to know which part of the code caused the problem. If I use @SuppressWarnings that part is in stealth mode, and an innocent part of the code will get the blame when it uses the wrong-typed value.

Remember, however, that cast and assertNonNull are not the solution, those are workarounds. Solutions must explicitly perform the check and provide application specific behavior to both outcomes of the check. Just as a cast without an instanceof check is still a land-mine, so is the use of the above helper: NPE can still occur. If you need to dereference a variable that’s not @NonNull you should really ask yourself:

  • How can it happen that I end up with a null value in this position?
  • How can the application safely and soundly continue in that situation?

These questions cannot be answered by any tool, these relate to the design of your software.

Help the JDT compiler helping you

This post showed you two things you can and should do to help the compiler helping you:

Add null annotations to resolve the contradiction that’s inherent in Java’s type system: a type can only either contain the value null or not contain the value null. Still Java’s type system opportunistically assumes a little bit of both. With annotations you can resolve the ambiguity and state which of the two possible types you mean.

Second, listen to the new type error messages. They’re fundamental to the analysis. If you disregard (or even disable) these messages there’s no point in letting the analysis apply all its sophisticated machinery. From false assumptions we cannot conclude anything useful.

If you apply these two hints, the compiler will be your friend and report quite some interesting findings. For a project that uses null annotations right from the first line of code written, this advice should be enough. The difficult part is: if you have a large existing code base already, the compiler will have a lot to complain. Think of migrating a fully untyped program to Java. You bet you could use some more help here. Let’s talk about that in future posts.

Written by Stephan Herrmann

February 21, 2012 at 20:39

Posted in Eclipse

Tagged with , , , ,

Help the JDT Compiler helping you! – 2: Resource leaks – continued

with one comment

In my previous post I showed the basics of a new analysis that I originally introduced in the JDT compiler as of 3.8 M3 and improved for M5. This post will give yet more insight into this analysis, which should help you in writing code that the compiler can understand.

Flow analysis – power and limitation

An advantage of implementing leak analysis in the compiler lies in the synergy with the existing flow analysis. We can precisely report whether a resource allocation is definitely followed by a close() or if some execution paths exist, where the close() call is by-passed or an early exit is taken (return or due to an exception). This is pretty cool, because it shows exactly those corner cases in your implementation, that are so easy to miss otherwise.

However, this flow analysis is only precise if each resource is uniquely bound to one local variable. Think of declaring all resource variables as final. If that is possible, our analysis is excellent, if you have multiple assignments to the same variable, if assignments happen only on some path etc, then our analysis can only do a best-effort attempt at keeping track of your resources. As a worst case consider this:

  Reader r = new FileReader(f);
  Reader r2 = null;
  while (goOn()) {
     if(hasMoreContent(r)) {
        readFrom(r);
     } else {
        r.close(); // close is nice, but which resource exactly is being closed??
     }
     if (maybe()) {
        r2 = r;
     }             // at this point: which resource is bound to r2??
     if (hasMoreFiles()) {
        r = new FileReader(getFile()); // wow, we can allocate plenty of resources in a loop
     }
  }
  if (r2 != null)
    r2.close();
 

This code may even be safe, but there’s no way our analysis can keep track of how many resources have been allocated in the loop, and which of these resources will be closed. Which one is the resource flowing into r2 to be closed at the end? We don’t know. So if you want the compiler to help you, pretty please, avoid writing this kind of code ๐Ÿ™‚

So what rules should you follow to get on terms with the compiler? To understand the mentioned limitation it helps to realize that our analysis is mostly connected to local variables, keeping some status bits for each of them. However, when analyzing variables the analysis has no notion of values, i.e., in the example the compiler can only see one variable r where at runtime an arbitrary number of Reader instances will be allocated, bound and dropped again.

Still, there are three special situations which the analysis can detect:

  Reader r = new FileReader("someFile");
  r = new FileReader("otherFile");
  r = new BufferedReader(r);
  Reader r2 = getReader();
  if (r2 != null) {
     r2.close();
  }
 
  1. In line 2 we’re leaking the instance from line 1, because after the assignment we no longer have a reference to the first reader and thus we cannot close it.
  2. However, line 3 is safe, because the same reader that is being dropped from r is first wrapped into a new BufferedReader and indirectly via that wrapper it is still reachable.
  3. Finally at the end of the example snippet, the analysis can see that r2 is either null or closed, so all is safe.

You see the compiler understands actually a lot of the semantics.

My fundamental advice is:

If the compiler warns about leaking resources and if you think the warning is unnecessary, try to better explain why you think so, first of all by using exactly one local variable per resource.

Resource ownership

Still not every method lacking a close() call signifies a resource leak. For an exact and definite analysis we would need one more piece of information: who owns any given resource?

Consider a group of methods happily passing around some resources among themselves. For them the same happens as for groups of people: diffusion of responsibility:

Well, no, I really thought that you were going to close this thing?!!?”.

If we had a notion of ownership we’d simple require the unique owner of each resource to eventually close that resource. However, such advanced concepts, while thoroughly explored in academia, are lacking from Java. To mitigate this problem, I made the following approximations of an ownership model:

  1. If a method allocates a resource, it owns it – initially.
  2. If a method obtains a resource by calling another method, it may potentially be responsible, since we cannot distinguish ownership from lending
  3. If a method passes a resource as an argument to another method (or constructor), it may or may not transfer ownership by this call.
  4. If a method receives a resource as a parameter, it assumes the caller is probably still responsible
  5. If a method passes a resource as its return value back to the caller, it rejects any responsibility
  6. If a resource is ever stored in a field, no single method feels responsible.
  7. If a resource is wrapped in an array we can no longer track the resource, but maybe the current method is still responsible?

In this list, green means: the compiler is encouraged to report anything fishy as a bug. Blue means, we still do the reporting, but weaken the message by saying “Potential resource leak”. Red means, the compiler is told to shut up because this code could only be checked by whole system analysis (which is not feasible for an incremental compiler).

The advice that follows from this is straight-forward:

Keep the responsibility for any resource local.
Do not pass it around and don’t store it in fields.
Do not talk to any strangers about your valuable resources!

In this regard, unclean code will actually cancel the leak analysis. If ownership of a resource is unclear, the compiler will just be quiet. So, do you think we should add a warning to signal whenever this happens? Notably, a warning when a resource is stored in a field?

The art of being quiet

Contrary to naive thinking, the art of good static analysis is not in reporting many issues. The art is in making yourself heard. If the compiler just rattles on with lots of uninteresting findings, no one will listen, no one will have the capacity to listen to all that.

A significant part of the work on resource leak analysis has gone into making the compiler quieter. And of course this is not just a matter of turning down the volume, but a matter of much smarter judgment of what the user might be interested in hearing.

By way of two recently resolved bugs (358903 and 368546) we managed to reduce the number of resource leak warnings reported against the sources of the Eclipse SDK from almost 100 down to 8. Calling this a great success may sound strange at first, but that it is.

At the level we reached now, I can confidently encourage everybody to enable this analysis (my recommendation: resource leaks = error, potential resource leaks = warning). The “Resource leak” problems indeed deserve a closer look, and also the potential ones could give valuable hints.

For each issue reported by the compiler you have three options:

  1. Agree that this is a bug
  2. Explain to the compiler why you believe the code is safe (unique assignment to locals, less passing around)
  3. Add @SuppressWarnings("resource") to tell the compiler that you know what you are doing.

But remember the nature of responsibility: if you say you don’t want to here any criticism you’d better be really sure. If you say you take the responsibility the compiler will be the humble servant who quietly forgets all worries.

Finally, if you are in the lucky position to use Java 7 for your projects, do the final step: enable “Resource not managed with try-with-resource” analysis. This was actually the start of this long journey: to let the compiler give hints where this new syntax would help to make your code safer and to make better visible why it is safe – with respect to resource leaks.

Final note: one of the bugs mentioned above was only resolved today. So with M5 you will still see some avoidable false positives. The next build from now should be better ๐Ÿ™‚

I’ll be back, soon, with more on our favorite exception: NPE.

Written by Stephan Herrmann

February 4, 2012 at 21:11

Posted in Eclipse

Tagged with , , ,

Help the JDT Compiler helping you! – 1: Resource Leaks

with 6 comments

During the Juno cycle a lot of work in the JDT has gone into more sophisticated static analysis, and some more is still in the pipe-line. I truly hope that once Juno is shipped this will help all JDT users to find more bugs immediately while still typing. However, early feedback regarding these features shows that users are starting to expect miracles from the analysis ๐Ÿ™‚

On the one hand seeing this is flattering, but on the other hand it makes me think we should perhaps explain what exactly the analysis can see and what is beyond its vision. If you take a few minutes learning about the concepts behind the analysis you’ll not only understand its limitations, but more importantly you will learn how to write code that’s better readable – in this case for reading by the compiler. Saying: with only slightly rephrasing your programs you can help the compiler to better understand what’s going on, to the effect that the compiler can answer with much more useful error and warning messages.

Since there’s a lot of analysis in this JDT compiler I will address just one topic per blog post. This post goes to improvements in the detection of resource leaks.

Resource leaks – the basics

Right when everybody believed that Eclipse Indigo RC 4 was ready for the great release, another blocker bug was detected: a simple resource leak basically prevented Eclipse from launching on a typical Linux box if more than 1000 bundles are installed. Coincidentally, at the same time the JDT team was finishing up work on the new try-with-resources statement introduced in Java 7. So I was thinking: shouldn’t the compiler help users to migrate from notoriously brittle handling of resources to the new construct that was designed specifically to facilitate a safe style of working with resources?

What’s a resource?

So, how can the compiler know about resources? Following the try-with-resources concept, any instance of type java.lang.AutoCloseable is a resource. Simple, huh? In order to extend the analysis also to pre Java 7 code, we also consider java.io.Closeable (available since 1.5).

Resource life cycle

The expected life cycle of any resource is : allocate—use—close. Simple again.

From this we conclude the code pattern we have to look for: where does the code allocate a closeable and no call to close() is seen afterwards. Or perhaps a call is seen but not all execution paths will reach that call, etc.

Basic warnings

With Juno M3 we released a first analysis that could now tell you things like:

  • Resource leak: “input” is never closed
  • Resource leak: “input” is never closed at this location (if a method exit happens before reaching close())

If the problem occurs only on some execution paths the warnings are softened (saying “potential leak” etc.).

Good, but

Signal to noise – part 1

It turned out that the analysis was causing significant noise. How come? The concepts are so clear and all code that wouldn’t exhibit the simple allocate—use—close life cycle should indeed by revised, shouldn’t it?

In fact we found several patterns, where these warnings were indeed useless.

Resource-less resources

We learned that not every subtype of Closeable really represents a resource that needs leak prevention. How many times have you invoked close() on a StringWriter, e.g.? Just have a look at its implementation and you’ll see why this isn’t worth the effort. Are there more classes in this category?

Indeed we found a total of 7 classes in java.io that purely operate on Java objects without allocating any resources from the operating system:

  • StringReader
  • StringWriter
  • ByteArrayInputStream
  • ByteArrayOutputStream
  • CharArrayReader
  • CharArrayWriter
  • StringBufferInputStream

For none of these does it make sense to warn about missing close().

To account for these classes we simply added a white list: if a class is in the list suppress any warnings/errors. This white list consists of exactly those 7 classes listed above. Sub-classes of these classes are not considered.

Wrapper resources

Another group of classes implementing Closeable showed up, that are not strictly resources themselves. Think of BufferedInputStream! Does it need to be closed?

Well? What’s your answer? The correct answer is: it depends. A few examples:

        void wrappers(String content) throws IOException {
                Reader r1, r2, r3, r4;
                r1 = new BufferedReader(new FileReader("someFile"));
                r2 = new BufferedReader(new StringReader(content));
                r3 = new FileReader("somefile");
                r4 = new BufferedReader(r3);
                r3.close();
        }
 

How many leaks? With same added smartness the compiler will signal only one resource leak: on r1. All others are safe:

  • r2 is a wrapper for a resource-less closeable: no OS resources are ever allocated here.
  • r3 is explicitly closed
  • r4 is just a wrapper around r3 and since that is properly closed, r4 does not hold onto any OS resources at the end.
  • returning to r1, why is that a leak? It’s a wrapper, too, but now the underlying resource (a FileReader) is not directly closed so it’s the responsibility of the wrapper and can only be triggered by calling close() on the wrapper r1.

EDIT: We are not recommending to close a wrapped resource directly as done with r3, closing the wrapper (r4) is definitely cleaner, and when wrapping a FileOutputStream with a BufferedOutputStream closing the former is actually wrong, because it may lose buffered content that hasn’t been flushed. However, the analysis is strictly focused on resource leaks and for analysing wrappers we narrow that notion to leaks of OS resources. For the given example, reporting a warning against r4 would be pure noise.

Summarizing: wrappers don’t directly hold an OS resource, but delegate to a next closeable. Depending on the nature and state of the nested closeable the wrapper may or may not be responsible for closing. In arbitrary chains of wrappers with a relevant resource at the bottom, closing any closeable in the chain (including the bottom) will suffice to release the single resource. If a wrapper chain is not properly closed the problem will be flagged against the outer-most wrapper, since calling close() at the wrapper will be delegated along all elements of the chain, which is the cleanest way of closing.

Also for wrappers the question arises: how does the compiler know? Again we set up a white list with all wrapper classes we found in the JRE: 20 classes in java.io, 12 in java.util.zip and 5 in other packages (the full lists are in TypeConstants.java, search for “_CLOSEABLES”).

Status and outlook

Yes, a leak can be a stop-ship problem.

Starting with Juno M3 we have basic analysis of resource leaks; starting with Juno M5 the analysis uses the two white lists mentioned above: resource-less closeables and resource wrappers. In real code this significantly reduces the number of false positives, which means: for the remaining warnings the signal-to-noise ratio is significantly better.

M5 will actually bring more improvements in this analysis, but that will be subject of a next post.

Written by Stephan Herrmann

January 26, 2012 at 15:25

Posted in Eclipse, Uncategorized

Tagged with , , ,

Object Teams with Null Annotations

with 5 comments

The recent release of Juno M4 brought an interesting combination: The Object Teams Development Tooling now natively supports annotation-based null analysis for Object Teams (OT/J). How about that? ๐Ÿ™‚
NO NPE

The path behind us

Annotation-based null analysis has been added to Eclipse in several stages:

Using OT/J for prototyping
As discussed in this post, OT/J excelled once more in a complex development challenge: it solved the conflict between extremely tight integration and separate development without double maintenance. That part was real fun.
Applying the prototype to numerous platforms
Next I reported that only one binary deployment of the OT/J-based prototype sufficed to upgrade any of 12 different versions of the JDT to support null annotations — looks like a cool product line
Pushing the prototype into the JDT/Core
Next all of the JDT team (Core and UI) invested efforts to make the new feature an integral part of the JDT. Thanks to all for this great collaboration!
Merging the changes into the OTDT
Now, that the new stuff was mixed back into the plain-Java implementation of the JDT, it was no longer applicable to other variants, but the routine merge between JDT/Core HEAD and Object Teams automatically brought it back for us. With the OTDT 2.1 M4, annotation-based null analysis is integral part of the OTDT.

Where we are now

Regarding the JDT, others like Andrey, Deepak and Aysush have beaten me in blogging about the new coolness. It seems the feature even made it to become a top mention of the Eclipse SDK Juno M4. Thanks for spreading the word!

Ah, and thanks to FOSSLC you can now watch my ECE 2011 presentation on this topic.

Two problems of OOP, and their solutions

Now, OT/J with null annotations is indeed an interesting mix, because it solves two inherent problems of object-oriented programming, which couldn’t differ more:

1.: NullPointerException is the most widespread and most embarrassing bug that we produce day after day, again and again. Pushing support for null annotations into the JDT has one major motivation: if you use the JDT but don’t use null annotations you’ll no longer have an excuse. For no good reasons your code will retain these miserable properties:

  • It will throw those embarrassing NPEs.
  • It doesn’t tell the reader about fundamental design decisions: which part of the code is responsible for handling which potential problems?

Why is this problem inherent to OOP? The dangerous operator that causes the exception is this:

right, the tiny little dot. And that happens to be the least dispensable operator in OOP.

2.: Objectivity seems to be a central property on any approach that is based just on Objects. While so many other activities in software engineering are based on the insight that complex problems with many stakeholders involved can best be addressed using perspectives and views etc., OOP forces you to abandon all that: an object is an object is an object. Think of a very simple object: a File. Some part of the application will be interested in the content so it can decode the bytes and do s.t. meaningful with it, another part of the application (maybe an underlying framework) will mainly be interested in the path in the filesystem and how it can be protected against concurrent writing, still other parts don’t care about either but only let you send the thing over the net. By representing the “File” as an object, that object must have all properties that are relevant to any part of the application. It must be openable, lockable and sendable and whatnot. This yields
bloated objects and unnecessary, sometimes daunting dependencies. Inside the object all those different use cases it is involved in can not be separated!

With roles objectivity is replaced by a disciplined form of subjectivity: each part of the application will see the object with exactly those properties it needs, mediated by a specific role. New parts can add new properties to existing objects — but not in the unsafe style of dynamic languages, but strictly typed and checked. What does it mean for practical design challenges? E.g, direct support for feature oriented designs – the direct path to painless product lines etc.

Just like the dot, objectivity seems to be hardcoded into OOP. While null annotations make the dot safe(r), the roles and teams of OT/J add a new dimension to OOP where perspectives can be used directly in the implementation. Maybe it does make sense, to have both capabilities in one language ๐Ÿ™‚ although one of them cleans up what should have been sorted out many decades ago while the other opens new doors towards the future of sustainable software designs.

The road ahead

The work on null annotations goes on. What we have in M4 is usable and I can only encourage adopters to start using it right now, but we still have an ambitious goal: eventually, the null analysis shall not only find some NPEs in your program, but eventually the absense of null related errors and warnings shall give the developer the guarantee that this piece of code will never throw NPE at runtime.

What’s missing towards that goal:

  1. Fields: we don’t yet support null annotations for fields. This is next on our plan, but one particular issue will require experimentation and feedback: how do we handle the initialization phase of an object, where fields start as being null? More on that soon.
  2. Libraries: we want to support null specifications for libraries that have no null annotations in their source code.
  3. JSR 308: only with JSR 308 will we be able to annotate all occurrences of types, like, e.g., the element type of a collection (think of List)

Please stay tuned as the feature evolves. Feedback including bug reports is very welcome!

Ah, and one more thing in the future: I finally have the opportunity to work out a cool tutorial with a fellow JDT committer: How To Train the JDT Dragon with Ayushman. Hope to see y’all in Reston!

Written by Stephan Herrmann

December 20, 2011 at 22:32

“OT/J 7”

leave a comment »

In my previous post I wrote about the challenge of maintaining 12 variants of the JDT/Core to support any combination of Java 6 vs. 7, Java vs. OT/J, with or without support for null annotations at versions Indigo GA, SR1 and Juno M1.

Meanwhile I played a bit with the combination of OT/J with Java 7. While I’m not that excited about Java 7 I found a way to combine the two that I found quite cute.

The typical OT/J foo-bar story

Consider the following classes:

public class Foo {
    public void foo() { System.out.println("foo"); }
}
public team class MyTeam {
    protected class R playedBy Foo {
        callin void bar() { System.out.println("bar"); }
        bar <- replace foo;
    }
}
 

So, we can use the team to replace the output “foo” by the output “bar”.
Now, a client calls f.foo() every now and then and may want to use the team to turn just one of the foos into bar. He may try this:

    public static void main(String[] args) {
        Foo f = new Foo();
        f.foo();
        test(f);
        f.foo();
    }
    static void test(Foo f) {
        MyTeam aTeam = new MyTeam();
        aTeam.activate();
        f.foo();
    }
 

He may expect this output:

foo
bar
foo

BUT, that’s not what he gets: the team remains active and we’ll see

foo
bar
bar

BUMMER.

What is a resource?

I kind-of like the new try-with-resources construct, so what can we actually use it for, is it just for FileReaders and things like that?

Since a resource is anything that is a subtype of AutoCloseable we can use the concept for anything that has an open-close lifecycle. So, what about — teams? Well, teams don’t have open() and close() methods, but isn’t team activation related? Activation opens a context and deactivation closes the context, no?

Let’s turn the team into an AutoCloseable:

public team class MyTeam implements AutoCloseable {
    // role R as before ...
 
    /** Just a simple factory method for active teams. */
    public static MyTeam createActiveTeam() {
        MyTeam t = new MyTeam();
        t.activate();
        return t;
    }
 
    @Override
    public void close() { // please don't throw anything here, it spoils the whole story!
        deactivate();
    }
}
 

Compiler-guided development

Before jumping at the solution, let me show you, what the new warning from Bug 349326 will tell you:

Ah, this is when our developer wakes up: “I might have to close that thing, good, I’m smart enough to do that”. Unfortunately, our developer is too smart, relying on some magic of the hashCode of his Foo:

You imagine, that now the full power of the compiler’s flow analysis will watch whether the thing is really closed on every path!

But even when the dev stops trying to outsmart the compiler, the compiler isn’t 100% happy:

We see the desired output now, but the method can and probably should still be improved: tell us what you mean: are these just three independent method calls? No, lines 1 and 3 are intended as a bracket for line 2: only within this context should aTeam affect the behavior of f.

Here’s our brave new OT/J 7 world:

Method test2() normally behaves identical to test1(), but we have two important differences:

  • test2() is safe against exceptions, the team will always be deactivated
  • test2() makes the intention explicit: the try () { } construct is not only a lexical block, but also a well defined semantic context: telling us that only inside this context aTeam is open/active, and outside it’s not. We don’t need to sequentially read the statements of the method to learn this.

An essential extension

Following my argumentation, OT/J must have been waiting for this language extension for a long time, no? Well, not exactly waiting. In fact, here’s something we had from the very beginning almost a decade ago:

But still, it’s nice to see that now every developer can achieve the same effect for different applications, beyond input streams and teams ๐Ÿ™‚

Here’s an exercise for the reader: the within construct in OT/J activates the team only for the current thread. With try-with-resources you can easily setup a context for global team activation and deactivation.

Happy hacking!

Written by Stephan Herrmann

August 25, 2011 at 18:32

Posted in Eclipse, Object Teams

Tagged with , , , ,

Mix-n-match language support

leave a comment »

I’ve been involved in the release of different versions of the JDT lately, supporting different flavors of Java.

Classical release management

At the core we have the plain JDT, of which we published the 3.7.0 release in June and right now first release candidates are being prepared towards the 3.7.1 service release, which will be the first official release to support Java 7. At the same time the first milestones towards 3.8 are being built. OK, this is almost normal business — with the exception of the service release differs more than usual from its base release, due to the unhappy timing of the release of Java 7 vs. Eclipse 3.7.

So that’s 3 versions in 2 month’s time.

First variant: Object Teams

The same release plan is mirrored by the Object Teams releases 2.0.0, 2.0.1RC1, 2.1.0M1. Merging the delta from JDT 3.7 to 3.7.1 into the OTDT was a challenge, given that this delta contained the full implementation of all that’s new in Java 7. Still with the experience of regularly merging JDT/Core changes into the OT variant, the pure merging was less than one day plus a couple more days until all 50000+ tests were green again. The nice thing about the architecture of the OTDT: after merging the JDT/Core, I was done. Since all other adaptations of the JDT are implemented using OT/Equinox adopting, e.g., all the new quick assists for Java 7 required a total of zero minutes integration time.

I took the liberty of branching 2.0.x and 2.1 only after integrating the Java 7 support, which also means that 2.1 M1 has only a small number of OT-specific improvements that did not already go into 2.0.1.

This gives 6 versions of the JDT in 2 month’s time.

Prototyping annotation based null analysis

As I wrote before, I’m preparing a comprehensive new feature for the JDT/Core: static analysis for potential NullPointerException based on annotations in the code. The latest patch attached to the bug had almost 3000 lines. Recent discussions at ECOOP made me change my mind in a few questions, so I changed some implementation strategies. Luckily the code is well modularized due to the use of OT/Equinox.

Now came the big question: against which version of the JDT should I build the null-annotation add-on? I mean, which of the 6 versions I have been involved in during the last 2 months?

As I like a fair challenge every now and then I decided: all six, i.e., I wanted to support adding the new static analysis to all six JDT versions mentioned before.

Integration details

Anybody who has worked on a Java compiler will confirm: if you change one feature of the compiler chances are that any other feature can be broken by the change (I’m just paraphrasing: “it’s complex”). And indeed, applying the nullity plug-in to the OTDT caused some headache at first, because both variants of the compiler make specific assumptions about the order in which specific information is available during the compilation process. It turned out that two of these assumptions where simply incompatible, so I had to make some changes (here I made the null analysis more robust).

At the point where I thought I was done, I tripped over an ugly problem that’s intrinsic to Java.
The nullity plug-in adapts a method in the JDT/Core which contains the following switch statement:

        while ((token = this.scanner.getNextToken()) != TerminalTokens.TokenNameEOF) {
                IExtendedModifier modifier = null;
                switch(token) {
                        case TerminalTokens.TokenNameabstract:
                                modifier = createModifier(Modifier.ModifierKeyword.ABSTRACT_KEYWORD);
                                break;
                        case TerminalTokens.TokenNamepublic:
                                modifier = createModifier(Modifier.ModifierKeyword.PUBLIC_KEYWORD);
                                break;
                        // more cases
                }
        }
 

I have a copy of this method where I only added a few lines to one of the case blocks.
Compiles fine against any version of the JDT. But Eclipse hangs when I install this plugin on top of a wrong JDT version. What’s wrong?

The problem lies in the (internal) interface TerminalTokens. The required constants TokenNameabstract etc. are of course present in all versions of this interface, however the values of these constants change every time the parser is generated anew. If constants were really abstractions that encapsulate their implementation values, all would be fine, but the Java byte code knows nothing about such an abstraction, all constant values are inlined during compilation. In other words: the meaning of a constant depends solely on the definitions which the compiler sees during compilation. Thus compiling the above switch statement hardcodes a dependency on one particular version of the interface TerminalTokens. BAD.

After recognizing the problem, I had to copy some different versions of the interface into my plug-in, implement some logic to translate between the different encodings and that problem was solved.

What’s next?

Nothing is next. At this point I could apply the nullity plug-in to all six versions of the JDT and all are behaving well.

We indeed have 12 versions of the JDT in 2 month’s time.

Mix-n-match

Would you like Java with our without the version 7 enhancements (stable release or milestone)? May I add some role and team classes? How about a dash more static analysis? It turns out we have more than just one product, we have a full little product line with features to pick or opt-out:

 

Java 6

Java 7
Indigo

Indigo SR1

Juno M1
no null annotations

Plain JDT

 

 

 
OTDT

 

 

 
with null annotations

Plain JDT

 

 

 
OTDT

 

 

 

Just make your choice ๐Ÿ™‚
Happy hacking with null annotations and try-with-resources in OT/J.

EclipseCon Europe 2011  

BTW: if you want to hear a bit more about the work on null annotations, you should really come to EclipseCon Europe — why not drop a comment at this submission ๐Ÿ™‚

Written by Stephan Herrmann

August 19, 2011 at 17:22

Null annotations: prototyping without the pain

with 3 comments


So, I’ve been working on annotations @NonNull and @Nullable so that the Eclipse Java compiler can statically detect your NullPointerExceptions already during compile time (see also bug 186342).

By now it’s clear this new feature will not be shipped as part of Eclipse 3.7, but that needn’t stop you from trying it, as I have uploaded the thing as an OT/Equinox plugin.

Behind the scenes: Object Teams


Today’s post shall focus on how I built that plugin using Object Teams, because it greatly shows three advantages of this technology:

  • easier maintenance
  • easier deployment
  • easier development


Before I go into details, let me express a warm invitation to our EclipseCon tutorial on Thursday morning. We’ll be happy to guide your first steps in using OT/J for your most modular, most flexible and best maintainable code.

Maintenance without the pain

It was suggested that I should create a CVS branch for the null annotation support. This is a natural choice, of course. I chose differently, because I’m tired of double maintenance, I don’t want to spend my time applying patches from one branch to the other and mending merge conflicts. So I avoid it wherever possible. You don’t think this kind of compiler enhancement can be developed outside the HEAD stream of the compiler without incurring double maintenance? Yes it can. With OT/J we have the tight integration that is needed for implementing the feature while keeping the sources well separated.

The code for the annotation support even lives in a different source repository, but the runtime effect is the same as if all this already were an integral part of the JDT/Core. Maybe I should say, that for this particular task the integration using OT/J causes a somewhat noticable performance penalty. The compiler does an awful lot of work and hooking into this busy machine comes at a price. So yes, at the end of the day this should be re-integrated into the JDT/Core. But for the time being the OT/J solution well serves its purpose (and in most other situations you won’t even notice any impact on performance plus we already have further performance improvements in the OT/J runtime in our development pipeline).

Independent deployment

Had I created a branch, the only way to get this to you early adopters would have been via a patch feature. I do have some routine in deploying patch features but they have one big drawback: they create a tight dependency to the exact version of the feature which you are patching. That means, if you have the habit of always updating to the latest I-build of Eclipse I would have to provide a new patch feature for each individual I-build released at Eclipse!

Not so for OT/Equinox plug-ins: in this particular case I have a lower bound: the JDT/Core must be from a build ≥ 20110226. Other than that the same OT/J-based plug-in seemlessly integrates with any Eclipse build. You may wonder, how can I be so sure. There could be changes in the JDT/Core that could break the integration. Theoretically: yes. Actually, as a JDT/Core committer I’ll be the first to know about those changes. But most importantly: from many years’ experience of using this technology I know such breakage is very seldom and should a problem occur it can be fixed in the blink of an eye.

As a special treat the OT/J-based plug-in can even be enabled/disabled dynamically at runtime. The OT/Equinox runtime ships with the following introspection view:

Simply unchecking the second item dynamically disables all annotation based null analysis, consistently.

Enjoyable development

The Java compiler is a complex beast. And it’s not exactly small. Over 5 Mbytes of source spread over 323 classes in 13 packages. The central package of these (ast) comprising no less than 109 classes. To add insult to injury: each line of this code could easily get you puzzling for a day or two. It ain’t easy.

If you are a wizard of the patches feel free to look at the latest patch from the bug. Does that look like s.t. you’d like to work on? Not after you’ve seen how nice & clean things can be, I suppose.

First level: overview

Instead of unfolding the package explorer until it shows all relevant classes (at what time the scrollbar will probably be too small to grasp) a quick look into the outline suffices to see everything relevant:

Here we see one top-level class, actually a team class. The class encapsulates a number of role classes containing the actual implementation.

Navigation to the details

Each of those role classes is bound to an existing class of the compiler, like:

   protected class MessageSend playedBy MessageSend { ...
(Don’t worry about identical names, already from the syntax it is clear that the left identifier MessageSend denotes a role class in the current team, whereas the second MessageSend refers to an existing base class imported from some other package).

Ctrl-click on the right-hand class name takes you to that base class (the packages containing those base classes are indicated in the above screenshot). This way the team serves as the single point of reference from which each affected location in the base code can be reached with a single mouse click – no matter how widely scattered those locations are.

When drilling down into details a typical roles looks like this:

The 3 top items are  “callout” method bindings providing access to fields or methods of the base object. The bottom item is a regular method implementing the new analysis for this particular AST node, and the item above it defines a  “callin” binding which causes the new method to be executed after each execution of the corresponding base method.

Locality of new information flows

Since all these roles define almost normal classes and objects, additional state can easily be introduced as fields in role classes. In fact some relevant information flows of the implementation make use of role fields for passing analysis results directly from one role to another, i.e., the new analysis mostly happens by interaction among the roles of this team.

Selective views, e.g., on inheritance structures

As a final example consider the inheritance hierarchy of class Statement: In the original this is a rather large tree:

Way too large actually to be fully unfolded in a single screenshot. But for the implementation at hand most of these
classes are irrelevant. So at the role layer we’re happy to work with this reduced view:


This view is not obtained by any filtering in the IDE, but that’s indeed the real full inheritance tree of the role class Statement. This is just one little example of how OT/J supports the implementation of selective views. As a result, when developing the annotation based null analysis, the code immediately provides a focused view of everything that is relevant, where relevance is directly defined by design intention.

A tale from the real world

I hope I could give an impression of a real world application of OT/J. I couldn’t think of a nicer structure for a feature of this complexity based on an existing code base of this size and complexity. Its actually fun to work with such powerful concepts.

Did I already say? Don’t miss our EclipseCon tutorial ๐Ÿ™‚

Hands-on introduction to Object Teams

See you at

Written by Stephan Herrmann

March 14, 2011 at 00:18