The Object Teams Blog

Adding team spirit to your objects.

Help the JDT Compiler helping you! – 1: Resource Leaks

with 6 comments

During the Juno cycle a lot of work in the JDT has gone into more sophisticated static analysis, and some more is still in the pipe-line. I truly hope that once Juno is shipped this will help all JDT users to find more bugs immediately while still typing. However, early feedback regarding these features shows that users are starting to expect miracles from the analysis 🙂

On the one hand seeing this is flattering, but on the other hand it makes me think we should perhaps explain what exactly the analysis can see and what is beyond its vision. If you take a few minutes learning about the concepts behind the analysis you’ll not only understand its limitations, but more importantly you will learn how to write code that’s better readable – in this case for reading by the compiler. Saying: with only slightly rephrasing your programs you can help the compiler to better understand what’s going on, to the effect that the compiler can answer with much more useful error and warning messages.

Since there’s a lot of analysis in this JDT compiler I will address just one topic per blog post. This post goes to improvements in the detection of resource leaks.

Resource leaks – the basics

Right when everybody believed that Eclipse Indigo RC 4 was ready for the great release, another blocker bug was detected: a simple resource leak basically prevented Eclipse from launching on a typical Linux box if more than 1000 bundles are installed. Coincidentally, at the same time the JDT team was finishing up work on the new try-with-resources statement introduced in Java 7. So I was thinking: shouldn’t the compiler help users to migrate from notoriously brittle handling of resources to the new construct that was designed specifically to facilitate a safe style of working with resources?

What’s a resource?

So, how can the compiler know about resources? Following the try-with-resources concept, any instance of type java.lang.AutoCloseable is a resource. Simple, huh? In order to extend the analysis also to pre Java 7 code, we also consider java.io.Closeable (available since 1.5).

Resource life cycle

The expected life cycle of any resource is : allocate—use—close. Simple again.

From this we conclude the code pattern we have to look for: where does the code allocate a closeable and no call to close() is seen afterwards. Or perhaps a call is seen but not all execution paths will reach that call, etc.

Basic warnings

With Juno M3 we released a first analysis that could now tell you things like:

  • Resource leak: “input” is never closed
  • Resource leak: “input” is never closed at this location (if a method exit happens before reaching close())

If the problem occurs only on some execution paths the warnings are softened (saying “potential leak” etc.).

Good, but

Signal to noise – part 1

It turned out that the analysis was causing significant noise. How come? The concepts are so clear and all code that wouldn’t exhibit the simple allocate—use—close life cycle should indeed by revised, shouldn’t it?

In fact we found several patterns, where these warnings were indeed useless.

Resource-less resources

We learned that not every subtype of Closeable really represents a resource that needs leak prevention. How many times have you invoked close() on a StringWriter, e.g.? Just have a look at its implementation and you’ll see why this isn’t worth the effort. Are there more classes in this category?

Indeed we found a total of 7 classes in java.io that purely operate on Java objects without allocating any resources from the operating system:

  • StringReader
  • StringWriter
  • ByteArrayInputStream
  • ByteArrayOutputStream
  • CharArrayReader
  • CharArrayWriter
  • StringBufferInputStream

For none of these does it make sense to warn about missing close().

To account for these classes we simply added a white list: if a class is in the list suppress any warnings/errors. This white list consists of exactly those 7 classes listed above. Sub-classes of these classes are not considered.

Wrapper resources

Another group of classes implementing Closeable showed up, that are not strictly resources themselves. Think of BufferedInputStream! Does it need to be closed?

Well? What’s your answer? The correct answer is: it depends. A few examples:

        void wrappers(String content) throws IOException {
                Reader r1, r2, r3, r4;
                r1 = new BufferedReader(new FileReader("someFile"));
                r2 = new BufferedReader(new StringReader(content));
                r3 = new FileReader("somefile");
                r4 = new BufferedReader(r3);
                r3.close();
        }
 

How many leaks? With same added smartness the compiler will signal only one resource leak: on r1. All others are safe:

  • r2 is a wrapper for a resource-less closeable: no OS resources are ever allocated here.
  • r3 is explicitly closed
  • r4 is just a wrapper around r3 and since that is properly closed, r4 does not hold onto any OS resources at the end.
  • returning to r1, why is that a leak? It’s a wrapper, too, but now the underlying resource (a FileReader) is not directly closed so it’s the responsibility of the wrapper and can only be triggered by calling close() on the wrapper r1.

EDIT: We are not recommending to close a wrapped resource directly as done with r3, closing the wrapper (r4) is definitely cleaner, and when wrapping a FileOutputStream with a BufferedOutputStream closing the former is actually wrong, because it may lose buffered content that hasn’t been flushed. However, the analysis is strictly focused on resource leaks and for analysing wrappers we narrow that notion to leaks of OS resources. For the given example, reporting a warning against r4 would be pure noise.

Summarizing: wrappers don’t directly hold an OS resource, but delegate to a next closeable. Depending on the nature and state of the nested closeable the wrapper may or may not be responsible for closing. In arbitrary chains of wrappers with a relevant resource at the bottom, closing any closeable in the chain (including the bottom) will suffice to release the single resource. If a wrapper chain is not properly closed the problem will be flagged against the outer-most wrapper, since calling close() at the wrapper will be delegated along all elements of the chain, which is the cleanest way of closing.

Also for wrappers the question arises: how does the compiler know? Again we set up a white list with all wrapper classes we found in the JRE: 20 classes in java.io, 12 in java.util.zip and 5 in other packages (the full lists are in TypeConstants.java, search for “_CLOSEABLES”).

Status and outlook

Yes, a leak can be a stop-ship problem.

Starting with Juno M3 we have basic analysis of resource leaks; starting with Juno M5 the analysis uses the two white lists mentioned above: resource-less closeables and resource wrappers. In real code this significantly reduces the number of false positives, which means: for the remaining warnings the signal-to-noise ratio is significantly better.

M5 will actually bring more improvements in this analysis, but that will be subject of a next post.

Advertisements

Written by Stephan Herrmann

January 26, 2012 at 15:25

Posted in Eclipse, Uncategorized

Tagged with , , ,

6 Responses

Subscribe to comments with RSS.

  1. Closing wrappers is more complex than you may be thinking. Consider that many wrappers actually buffer content (most obviously the ones that start with “Buffered”)… If you don’t close (or flush) them, the content may not be passed to the wrapped stream.

    In your example, anything written to r4 gets buffered before being written to r3. If r4 isn’t flushed or closed, r3 will likely not have received all of the content that was written to r4 (unless the last data written to r4 happens to exactly hit the buffer boundry). Closing r3 effectively truncates the data.

    Please rethink the handling of stream decorators like these…

    Scott Stanchfield

    January 26, 2012 at 16:30

  2. @Scott: thanks, that is a very valid point.

    However, it is not a point to be considered for leak analysis, or am I missing anything?

    We don’t certify correct program behavior, just absence of a very specific kind of problem: resource leaks.

    Programs in general are much more complex than any static analysis can digest 🙂

    OTOH, your comment confirms our strategy to always flag resource problems against the top of a resource wrapper chain.

    stephan

    January 26, 2012 at 17:03

  3. While it may not be a resource leak issue, making the distinction in this manner is bound to lead to confusion…

    To the user, it seems like the compiler is telling him when he forgot to close a closeable, and they’ll likely think they have the “close” logic proper in that example.

    I agree that we can’t catch everything statically, but this seems like you’re making an exception in the more general analysis to rule _out_ a check that would be incredibly helpful.

    Scott Stanchfield

    January 27, 2012 at 02:24

  4. I added a paragraph for clarification about Buffered* and about the closing of wrapped resources.

    Actually, in the example the requested warning against r4 would be useless, because a BufferedReader cannot truncated any file.

    In my next post I’ll say more about the balance between reporting everything that looks fishy and being shy regarding any possible false positives.

    stephan

    February 4, 2012 at 12:18

  5. […] my previous post I showed the basics of a new analysis that I originally introduced in the JDT compiler as of 3.8 M3 […]

  6. […] my little excursion to detecting resource leaks let me return to our favorite bug: […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: