The Object Teams Blog

Adding team spirit to your objects.

Builds are like real software – or even more so

with 8 comments

Being a part-time release engineer for the Object Teams project I can only agree with every word Kim writes about the job, I wish I could hire her for our project 🙂

She writes:

“Nobody in needs to understand how the build works, they just need to push a button. That’s great. Until the day before a release when your build fails with a cryptic message about unresolved dependencies. And you have no idea how to fix it. And neither does anyone else on the team.”

That puts a sad smile on my face and I’d like to add a little quality metric that seems cruel for today’s build systems, but might actually be useful for any software:

No software can be better than its worst error message.

One extreme I experienced was in a PDE/Build-ant-build which I had to set to verbose to get any useful answer but then I had to find the relevant error message deeply buried in literally tens of megabytes of log output. Takes ages to browse that log file. Other tools rank towards the other end of the spectrum saying basically “it didn’t work”.

Why is the worst error message relevant? When you hit that worst message it’s close to saying “game over”. Especially when working on a build I’ve come to the point time and again where all my creativity and productivity came to a grinding halt and for days or weeks I simply made zero progress because I had no idea why that system didn’t work and what it expected me to do to fix the thing. Knock-out.

Obviously I hate that state when I make no progress towards my goal. And typically that state is reached by poor communication from some framework back to me.

Real coolness

I know people usually don’t like to work on improving error messages, but please, don’t think good error messages are any bit less cool than running your software on mars. On the one hand we try to build tools that improve developers’ productivity by a few percent and than the tool will give answers that bring that very productivity down to zero. That’s – inconsistent.

I’m tempted to repeat the p2 story here. Many will remember the merciless dump of data from the sat solver that p2 gave in its early days. Some will consider the problem solved by now. Judge for yourself: what’s the worst-case time a regular Eclipse user will need to understand what p2 is telling him/her by one of its error messages.

The intention of this post is not to blame any particular technology. The list would be quite long anyway. It’s about general awareness (big words, sorry 🙂 ).

Consider the worst case

Again, why worst case? Because the worst case will happen. And it’s enough if it hits you once to easily compensate all the time savings the tool otherwise brought to you.

Communicate!

Framework developers, tool smiths: let your software communicate with the user and let it be especially helpful when the user is in dire need of help.

One small contribution in this field I’d like to share with you: in the OTDT every error/warning raised by the compiler not only tries to precisely describe what’s wrong but it is directly linked to the corresponding paragraph in the language definition that is violated by the current code. At least this should completely explain why the current code is wrong. It’s a small step, but I feel a strong need for linking specific help to every error message.

But first, the software has to anticipate every single error that will occur in order to produce useful messages. That’s the real reason why creating complex software is so challenging. Be it a build system or the “real” software.

Be cool, give superb error messages!

Advertisements

Written by Stephan Herrmann

September 4, 2011 at 15:28

Posted in Eclipse, OTDT

Tagged with , , ,

8 Responses

Subscribe to comments with RSS.

  1. Great post Stephan! I like your point that no software can be better than it’s worst error message. A great point!

    Kim Moir

    September 5, 2011 at 21:10

  2. In order to improve the build process of a big RCP application, I tried to switch from a PDE Build based build setup to Tycho.

    Even if Tycho has a manageable and understandable “MANIFEST first” build process (compared to PDE Build), error messages related to dependency resolving and version checking are raised too lately to be of any help when you are in a hurry.

    PDE Build needs 20 minutes to compile my RCP application.
    Tycho needs 15 minutes just to read all the projects and check dependencies (with a single error message instead of one per unresolved dependency), and 40 minutes to complains about a version mismatch when about to compile the first plugin (but Maven allow to resume the build process to the failed project this time).

    Laurent Asfaux

    September 6, 2011 at 11:04

  3. @Kim: I’m glad you like it! 🙂

    @Laurent: From what you say, I’d conclude that for a build system excellent error messages are even more important than during normal development, because turn-around times are much higher: It’s not very productive if you have to wait an hour for reading: “it didn’t work”. At that time you want very precise information so you have a good chance that the next build will actually work. Trial and error involving hours of waiting is painful.

    I can also read your comment as saying: only quick error messages are good error messages, which is a good point, too.

    stephan

    September 6, 2011 at 12:21

  4. Totally agree with you on this one. We switched from PDE Build to Buckminster about a year ago and never been happier. Builds are faster and more manageable, but this came at the cost of deciphering error messages that Buckminster sometimes would throw at you. Of course once you would figure out with the help of others what the error messages mean everything make sense, but at that point went you don’t know what in the world that error means, your heart sinks a bit and you feel all alone. The good thing is that these errors are predictable after sometime, so once you’ve spent time figuring them out it gets easier, and writing “Errors FAQ” makes it easier for others.

    Alex Kravets

    September 6, 2011 at 17:09

  5. @Alex: Error FAQs indeed seem a viable workaround. Are you speaking of individual FAQs or are tool devs actually maintaining these?

    I guess the community can help in putting together a good FAQ, but still I’d like to push all tool devs to paying more attention to how the tool communicates with you. If they actively maintain the FAQ hopefully they also get ideas how to improve the error messages in the first place.

    stephan

    September 6, 2011 at 18:03

  6. @stephan For us (internal wiki) it is an FAQ where a particular error has explanation of error and possible solutions.
    Unfortunately this falls under documentation and many public projects lack good documentation, which means these lists a scattered over the web maintained by people who had dealt with these errors before.

    Alex Kravets

    September 6, 2011 at 18:27

  7. […] Kudos for this simple but oh so true proverb go to Stephan Herrmann. […]

  8. […] have been ranting about bad error messages, so in my own work, error messages better be helpful. At least I […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: