Compiling OT/Equinox projects using Tycho

In a previous post I showed how the tycho-compiler-jdt Maven plug-in can be used for compiling OT/J code with Maven.

Recently, I was asked how the same can be done for OT/Equinox projects. Given that we were already using parts from Tycho, this shouldn’t be so difficult, right?

Once you know the solution, finding the solution is indeed easy, also in this case. Here it is:

We use almost the same declaration as for plain OT/J applications:

    <pluginManagement>
	<plugin>
	    <groupId>org.eclipse.tycho</groupId>
	    <artifactId>tycho-compiler-plugin</artifactId>
	    <version>${tycho.version}</version>
	    
	    <dependencies>
		<dependency>
		    <groupId>org.eclipse.tycho</groupId>
		    <artifactId>tycho-compiler-jdt</artifactId>
		    <version>${tycho.version}</version>
		    <exclusions>
			
			<exclusion>
			    <groupId>org.eclipse.tycho</groupId>
			    <artifactId>org.eclipse.jdt.core</artifactId>
			</exclusion>
		    </exclusions>
		</dependency>
		<dependency>
		    
		    <groupId>org.eclipse</groupId>
		    <artifactId>objectteams-otj-compiler</artifactId>
		    <version>${otj.version}</version>
		</dependency>
	    </dependencies>
	</plugin>
    </pluginManagement>

So, what’s the difference? In both cases we need to adapt the tycho-compiler-jdt plug-in because that’s where we replace the normal JDT compiler with the OT/J variant. However, for plain OT/J applications tycho-compiler-jdt is pulled in as a dependency of maven-compiler-plugin and must be adapted on this path of dependencies, whereas in Tycho projects tycho-compiler-jdt is pulled in from tycho-compiler-plugin. Apparently, the exclusion mechanism is sensitive to how exactly a plug-in is pulled into the build. Interesting.

Once I figured this out, I created and published a new version of our Maven support for Object Teams: objectteams-parent-pom:2.1.1 — publishing Maven support for Object Teams 2.1.1 was overdue anyway 🙂

With the updated parent pom, a full OT/Equinox hello world pom now looks like this:

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
 
    <parent>
        
        <groupId>org.eclipse</groupId>
        <artifactId>objectteams-parent-pom</artifactId>
        <version>2.1.1</version>
    </parent>
 
    <artifactId>OTEquinox-over-tycho</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    
    <packaging>eclipse-plugin</packaging>
 
    <repositories>
 
        
        <repository>
            <id>ObjectTeamsRepository</id>
            <name>Object Teams Repository</name>
            <url>http://download.eclipse.org/objectteams/maven/3/repository</url>
        </repository>
 
        
        <repository>
            <id>Juno</id>
            <name>Eclipse Juno Repository</name>
            <url>http://download.eclipse.org/releases/juno</url>
            <layout>p2</layout>
        </repository>
    </repositories>
 
    <build>
        <plugins>
            <plugin>
                
                <groupId>org.eclipse.tycho</groupId>
                <artifactId>tycho-maven-plugin</artifactId>
                <extensions>true</extensions>
            </plugin>
        </plugins>
    </build>
</project>

Looks pretty straight forward, right?

To see the full OT/Equinox Hello World Example configured for Maven/Tycho simply import OTEquiTycho.zip as a project into your workspace.

cheers,
Stephan

Builds are like real software – or even more so

Being a part-time release engineer for the Object Teams project I can only agree with every word Kim writes about the job, I wish I could hire her for our project 🙂

She writes:

“Nobody in needs to understand how the build works, they just need to push a button. That’s great. Until the day before a release when your build fails with a cryptic message about unresolved dependencies. And you have no idea how to fix it. And neither does anyone else on the team.”

That puts a sad smile on my face and I’d like to add a little quality metric that seems cruel for today’s build systems, but might actually be useful for any software:

No software can be better than its worst error message.

One extreme I experienced was in a PDE/Build-ant-build which I had to set to verbose to get any useful answer but then I had to find the relevant error message deeply buried in literally tens of megabytes of log output. Takes ages to browse that log file. Other tools rank towards the other end of the spectrum saying basically “it didn’t work”.

Why is the worst error message relevant? When you hit that worst message it’s close to saying “game over”. Especially when working on a build I’ve come to the point time and again where all my creativity and productivity came to a grinding halt and for days or weeks I simply made zero progress because I had no idea why that system didn’t work and what it expected me to do to fix the thing. Knock-out.

Obviously I hate that state when I make no progress towards my goal. And typically that state is reached by poor communication from some framework back to me.

Real coolness

I know people usually don’t like to work on improving error messages, but please, don’t think good error messages are any bit less cool than running your software on mars. On the one hand we try to build tools that improve developers’ productivity by a few percent and than the tool will give answers that bring that very productivity down to zero. That’s – inconsistent.

I’m tempted to repeat the p2 story here. Many will remember the merciless dump of data from the sat solver that p2 gave in its early days. Some will consider the problem solved by now. Judge for yourself: what’s the worst-case time a regular Eclipse user will need to understand what p2 is telling him/her by one of its error messages.

The intention of this post is not to blame any particular technology. The list would be quite long anyway. It’s about general awareness (big words, sorry 🙂 ).

Consider the worst case

Again, why worst case? Because the worst case will happen. And it’s enough if it hits you once to easily compensate all the time savings the tool otherwise brought to you.

Communicate!

Framework developers, tool smiths: let your software communicate with the user and let it be especially helpful when the user is in dire need of help.

One small contribution in this field I’d like to share with you: in the OTDT every error/warning raised by the compiler not only tries to precisely describe what’s wrong but it is directly linked to the corresponding paragraph in the language definition that is violated by the current code. At least this should completely explain why the current code is wrong. It’s a small step, but I feel a strong need for linking specific help to every error message.

But first, the software has to anticipate every single error that will occur in order to produce useful messages. That’s the real reason why creating complex software is so challenging. Be it a build system or the “real” software.

Be cool, give superb error messages!

A Short Train Ride

Less than a week ago I happily announced that Object Teams is on the Indigo Train.
Much water has gone under the bridge since then and the above statement is history.

Events where triggered by what was actually a little bug in the b3 aggregator. Only by way of this bug some people noticed that there was a ”’patch feature”’ inside the repository, i.e., a feature (“Object Teams Patch for JDT/Core”) that replaces the jdt.core plugin with a variant.

One part of me is very happy this bug occurred because finally an issue got the attention I had tried to raise at various occasions before. The lesson is:

Installing stuff with p2 may do other things than just adding new functionality, it may change existing stuff.

(see this post, e.g.).

The other part of me got very worried because during that debate some harsh statements occurred that would effectively amount to excluding Object Teams from Eclipse.org. That’s a little more attention than I had intended. As in any heated debate some of the arguments sounded to me more like ideology than anything that could possibly be discussed open-mindedly.

I had mixed feelings regarding the technical scope: the outcry only banned one specific technology: patch features. I don’t see how the goal to protect a project’s bits and bytes against influence from other projects can be achieved without also discussing: access to internal, byte-code weaving and – worst of all, I believe – reflection. To be perfectly open: Object Teams uses all these techniques except for reflection. Personally, I would even argue for banning projects that do use setAccessible(true), but that’s not a realistic option because then quite likely the whole Train would dissolve into a mist.

I am actually guilty of a technical simplification during this debate: I focused too much on the idea that a user would explicitly select features to install, not accounting for the possibility that the jdt.core plugin can well be pulled in invisibly due to dependencies among plugins. So, yes, if Object Teams would still be in the repository, and if a user installed a package without the JDT and if that user did never select to install the JDT and if that user selects another feature that implicitly requires the jdt.core plugin, then that user would unexpectly install the OT variant of the jdt.core. I agree that this is not ideal. I personally would have been happy to take this risk because I know how thoroughly the OT variant is tested for compatibility. And for the remaining minuscule risk I would have been happy to promise same-day fixes. But risk assessment naturally depends on perspective and I understand that others come to different conclusions when weighing the issues.

From two days distance I can already laugh at one implication of the central argument, paraphrased as: the JDT/Core team must be protected against harmful actions from the OT team. When spelling this out in names, one of the sentences reads: “Stephan Herrmann must be protected against harmful actions by Stephan Herrmann”. I should really be careful, because I’ll never be able to escape him!

Where to go?

• As we’re banned from the Train, I willed hurriedly book a plane ticket to Indigo. Make sure we come by a Graduation office on the way.
• I do hope that bug 316702 get’s sufficient attention now. Seriously: if you are so detrimentally determined about patch features, then the UI must report it. And if it reports this, it might as well report other techniques that have similar effects!
• I appreciate any offers for helping OT/J towards a solution that avoids replacing a plugin. As of today and after more than seven years of looking at this, I see no way how this can be done, but that doesn’t mean we shouldn’t try still harder.

Unfortunately, the debate consumed all the time and energy I had planned for preparing a presentation at the EclipseCON Audition. However, Lynn finally made my day by letting me know that my submission is the lucky #42. So I’m making progress towards my all-seasons Eclipse collection 🙂

Let’s talk straight with our users

Users of Eclipse trust in its quality. But what happens when they install additional plug-ins that are not part of the original download package? We tend to say that plug-ins only add to the installed system, but by saying so we’re actually spreading a myth:

“Adding plug-ins to an Eclipse install does not affect anything that is already installed.”

This simplest way to falsify this myth is by installing two plug-ins that together produce a deadlock. Each plug-in in isolation may work just perfect, but adding the other will cause Eclipse to freeze.

No zero-risk software install

Sure, in the absence of any correctness proofs, we can never be sure that adding one more plug-in to an already complex system will not break anything. Concurrency is just one of the most cumbersome issues to get right. Other reasons exist, why installing more is not always better.

Install-time risk assement

Actually, the p2 UI already distinguishes two kinds of installs: signed and unsigned artifacts, and warns the user when unsigned content is about to be installed. This is great, because now the user can decide whether s/he accepts unsigned content or not.

But this question is just the tip of the iceberg.

Behind the scenes of p2

When you tell p2 to install a given feature a number of things can happen that a normal user will not be aware of:

• Additional required software may be pulled in, possibly from different projects / vendors / repositories
• New features may replace existing plug-ins (patch features).
• Installation may trigger touchpoint actions that modify central parts of the system
• Installed plug-ins may access non-public classes and members of existing plug-ins
• New plug-ins may hook into the Equinox framework (adaptor hooks)
• New plug-ins can weave into the bytecode of existing plug-ins.

Is this bad?

Don’t get me wrong: I believe all these ways of influencing the installation are very useful capabilities for specific goals (and Object Teams utilizes several of those). However, we may want to acknowledge that users have different goals. If you need to be 100% sure that installing a plug-in does not delete anything on your hard disk you will not be happy to learn about the touchpoint instruction “remove”, while apparently others think this a cool feature.

Touchpoint actions have no ethical value, neither good nor bad.

Negotiation over rules

Drawing the line of how a plug-in may affect the rest of the system can be done up-front, by defining rules of conduct. Thou shalt not use internal classes; thou shalt only add, never modify etc. But that would impose a one-size-fits-all regime over all plug-ins to be published, and installations that can or cannot be created from published plug-ins. This regime would never equally suite the anxious and the daring, users with highest security constraints and users loving to play around with the newest and coolest.

Better than one size would be two sizes (conservative & bleeding edge), but then we still wouldn’t meet en par with our users, with all their shades of goals (and fears). Instead of imposing rules I suggest to enter negotiation with our users. By that I meen:

• Really tell them what’s in the boxes
• Let them decide which boxes to install

Proposing: Install capabilities

I could think of a concept of install capabilities at the core of a solution:

• Each artifact may declare a set of non-standard capabilities required for installing, which could include all items from the above list (and more)
• During install a security/risk report will be presented to the user mentioning (a summary of) all capability requests
• Based on that report a user can select which requests to accept, possibly blocking risky plug-ins from installing
• The runtime should enforce that no feature/plug-in uses any install capabilities which it didn’t declare

In this context I previously coined the notion of Gradual Encapsulation including a more elaborate model of negotiation.

More specifically I filed bug 316702. Where I’d love to read some comments.

Are you going to Eclipse Summit Europe? Shouldn’t we talk about this in a BoF?

I strongly feel it’s time to talk straight to our users: adding new artifacts to an existing Eclipse install is not always purely additive. Installation can indeed modify existing parts. In an open-source community, we should tell our users what modifications are requested, what potential risks are associated with a given artifact. They are grown-up and able to decide for themselves.

Why I’m sometimes a bad bug reporter

OK, I use Eclipse for getting some work done. Eclipse is software so we know it contains bugs. Given that Eclipse is open source software, we all can only expect it to run smoothly if we diligently report back all errors we encounter (and provide steps on how to reproduce etc.). I know all that and I really want to be a good bug reporter because I really want a good experience using Eclipse because I really want to get the work done (and I may even want to sustain the impression that I’ve got everything under control and thus working with Eclipse is a delight).

A task

The other day, I was a very bad bug reporter, and only today I find some time to reason about what happened. This was my task: In preparing the initial contribution for the Object Teams Project I had to rename a bunch of things from org.objectteams to org.eclipse.objectteams. Simply, huh? Back in the days of emacs+bash I might have chosen to just use one big

find . -exec /bin/sed -i -e "s/org.objectteams/org.eclipse.objectteams/g" {} ;

and voila, if fortuna was with me, I might have been done at the return of that single command. But things were actually just a little bit more challenging, like, a few occurrences would have to remain unchanged plus while touching about every single file in our software I was going to also do some clean-up: rename some packages to foo.internal.bar, fixing some copyright headers, etc. Also I preferred to change one plug-in at a time which would mean that all references to plug-ins not yet processed should stay unchanged, too. Plus a few more little deviations from the grand search-and-replace-globally.

OK, since I like to see myself an Eclipse-wizard this was a nice challenge for its refactoring support. Plug-in by plug-in I renamed, I renamed packages with / or without subpackages, and after each step I wanted to see that compiler and PDE agree with all changes and signal that everything is still (or again) consistent, ready to be built, actually. Perhaps things started the get wrong when I estimated the effort as one or two hours. So, after a day or so, I wasn’t perfectly relaxed any more. My fault, should’ve known better about that estimate. BTW, one of the reasons it took so long was simply the size of my workspace in comparison to the power of my computer / hard-disk: every time I performed a rename with updates in non-Java files, I was nervously looking at the screen: “should I sit and wait for the preview page, or should I go to the kitchen, get a chocolate, coffee, just something?“. I did some emailing in parallel, but let’s just keep this: due to those response times
I wasn’t perfectly relaxed any more.

A story of bug reporting

What I was going to tell here is a story of bug reporting, because as a safe bet doing a real-life stress test to an Eclipse component should give you a good chance to discover and report a few bugs that have not yet been reported by others. And indeed, I was successful in discovering some bugs, in various components actually.

I think one of the first things that occurred was that the svn synchronize view would sometimes fail to open the compare editor, more precisely, I had to explicitly close the compare editor before comparing the next file. At first this really **** me off, because the error dialog was popping up in some kind of infinite loop. Fun!#$ Once I’d figure out how to work around this problem it soon became a habit to just close the compare editor before clicking the next. Next, the svn plugin made a refactoring fail, because it was trying to create a directory which the previous refactoring had already created. The most creative bug having to do with subversive was a chain-reaction of first failing to undo a refactoring and than during reporting this error blocking the UI of Eclipse so I could only kill the Eclipse process, insert a new coin and start a new game.

I don’t intend to blame a particular component. For clean-up of license headers I have a little home-grown plugin that I just wanted to quickly install into the running Eclipse instance, so I went for the cool new feature to export/install into the host. Oops, my plugin depends on another plugin that only exists in the workspace but not in the host, install failed for good reasons. I removed the dependency and tried again. Installation still failed for the same reason: the ghost of this removed dependency prevented installation into the host Eclipse. Oh, I should have incremented the version or let a version qualifier do this automatically, of course. Tried again, still failed. Tried something so slightly different I cannot recall, from there on it worked. Can I reproduce the two or three different levels of failure? I didn’t even take the time to think of it. Well I would’ve been disappointed without a bug from p2 in this list 😉 .

PDE did its share by reporting overlapping text edits in plugin.xml and therefore disabling its refactoring participant. What the **** caused those overlapping text edits, and how do I re-enable the refactoring participant to give it one more chance to behave well?

The list could go on if only I could remember. Instead I was happy to finish this 1.5 hours task after 2.7 days, ready to submit our initial code contribution, wow!

Looking back, I / we missed a great opportunity: we could have identified plenty of bugs in various components of Eclipse. With only a few more days of debugging I might have been able to present reproducing steps for all those bugs. And, if triaged and fixed by the corresponding devs, this might have contributed to M6 containing fewer of those bugs that just only occur in real world, never during testing. I failed, I managed only to submit two bug reports, with very little information on how to reproduce.

Lesson learned

Susan McCourt responded to an earlier bug report of mine in a very descriptive comment:

That is one of those things I’ve been meaning to fix forever, never wrote a
bug, and so keep forgetting to fix. And it seems like if I’m actually
[doing what triggers the bug], it’s because something is wrong, and so I again postpone
writing a bug.

Sure, when we hit a bug (or a bug hits us) we are always in some context of doing something challenging. Something that requires our mind to stay in focus. Something we want to get done.
Well, work isn’t perfectly linear, so we know how to react to interrupts. Bugs are such interrupts. Sometimes I like the challenge of isolating a bug etc. Sometimes I’m sufficiently relaxed when the bug occurs so I actually take the challenge. Sometimes the bug is sufficiently good-natured so making a small note and going back to the bug after the actual work is done is a perfect strategy. Some bugs, however, smell like depending on so many factors from your current context that reproduction an hour later seems extremely unlikely.

I think I have a solution to all this: given we don’t want to be distracted from our actual work, given also that some bugs need immediate care or they will escape our attempts to identify. Given some of the worst moments are when we start to isolate a bug and during that task a second bug stops us from working on the first bug etc. The only way to relentlessly follow all those tasks is to enable branching universes in your working environment. The frequent use of the work “task” may give a hint that I should finally start using Mylyn (I have no excuse for not doing so), but I would need a Mylyn that is able to capture full contexts: the complete state of my machine plus the full state of my brain. As a start I’m dreaming of always working in a virtual machine, and whenever something strange happens, I quickly (!!) create a snapshot of the virtual machine. Then I could first isolate (and fix 🙂 ) the bug that just occurred, and then go back to the exact point where I
interrupted my work and act as if nothing had happened. Branching universes with the ability of back porting fixes between branches is what I need. Of course the clock needs to be reset when returning from a bug reporting / fixing branch.

Yeah, that’s why I can’t quite live up to my dreams of good participation in open source development: I haven’t figured out how to enable branching universes for my character. If anybody has a hint on how to do this or any other strategy to not get overwhelmed between work and bug reporting, I’d really appreciate.

And if I don’t smile while writing a bug report, please excuse, I might just be terribly stressed because your bug interrupted my work on isolating another bug that stopped me from doing …

”Always look at the bright side of life…” 🙂