The Object Teams Blog

Adding team spirit to your objects.

Posts Tagged ‘build

Eclipse Neon.2 is on Maven Central

with 6 comments

It’s done, finally!

Bidding farewell to my pet peeve

In my job at GK Software I have the pleasure of developing technology based on Eclipse. But those colleagues consuming my technology work on software that has no direct connection to Eclipse nor OSGi. Their build technology of choice is Maven (without tycho that is). So whenever their build touches my technology we are facing a “challenge”. It doesn’t make a big difference if they are just invoking a code generator built using Xtext etc or whether some Eclipse technology should actually be included in their application runtime.

Among many troubles, I recall one situation that really opened my eyes: one particular build had been running successfully for some time, until one day it was fubar. One Eclipse artifact could no longer be resolved. Followed long nights of searching why that artifact may have disappeared, but we reassured ourselves, nothing had disappeared. Quite to the contrary somewhere on the wide internet (Maven Central to be precise) a new artifact had appeared. So what? Well, that artifact was the same that we also had on our internal servers. Well, if it’s the same, what’s the buzz? It turned out it had a one-char difference in its version: instead of 1.2.3.v20140815 its version was 1.2.3-v20140815. Yes take a close look, there is a difference. Bottom line, with both almost-identical versions available, Maven couldn’t figure out what to do, maybe each was considered as worse than the other, to the effect that Maven simply failed to use either. Go figure.

More stories like this and I realized that relying on Eclipse artifacts in Maven builds was always at the mercy of some volunteers, who typically don’t have a long-term relationship to Eclipse, who filled in a major gap by uploading individual Eclipse artifacts to Maven Central (thanks to you volunteers, please don’t take it personally: I’m happy that your work is no longer needed). Anybody who has ever studied the differences between Maven and OSGi (wrt dependencies and building that is) will immediately see that there are many possible ways to represent Eclipse artifacts (OSGi bundles) in a Maven pom. The resulting “diversity” was one of my pet peeves in my job.

At this point I decided to be the next volunteer who would screw up other people’s builds who would collaborate with the powers that be at Eclipse.org to produce the official uploads to Maven Central.

As of today, I can report that this dream has become reality, all relevant artifacts of Neon.2 that are produced by the Eclipse Project, are now “officially” available from Maven Central.

Bridging between universes

I should like to report some details of how our artifacts are mapped into the Maven world:

The main tool in this endeavour is the CBI aggregator, a model based tool for transforming p2 repositories in various ways. One of its capabilities is to create a Maven repository (a dual use repo actually, but the p2 side of this is immaterial to this story). That tool does a great job of extracting meta data from the p2 repo in order to create “meaningful” pom files, the key feature being: it copies all dependency information, which is originally authored in MANIFEST.MF, into corresponding declarations in the pom file.

Still a few things had to be settled, either by improving the tool, by fine tuning the input to the tool, or by some steps of post-processing the resulting Maven repo.

  • Group IDs
    While OSGi artifacts only have a single qualified Bundle-SymbolicName, Maven requires a two-part name: groupId x artifactId. It was easy to agree on using the full symbolic name for the artifactId, but what should the groups be? We settled on these three groups for the Eclipse Project:

    • org.eclipse.platform
    • org.eclipse.jdt
    • org.eclipse.pde
  • Version numbers
    In Maven land, release versions have three segments, in OSGi we maintain a forth segment (qualifier) also for releases. To play by Maven rules, we decided to use three-part versions for our uploads to Maven Central. This emphasizes the strategy to only publish releases, for which the first three parts of the version are required to be unique.
  • 3rd party dependencies
    All non-Eclipse artifacts that we depend on should be referenced by their proper coordinates in Maven land. By default, the CBI aggregator assigns all artifacts to the synthetic group p2.osgi.bundle, but if s.o. depends on p2.osgi.bundle:org.junit this doesn’t make much sense. In particular, it must be avoided that projects consuming Eclipse artifacts will get the same 3rd party library under two different names (perhaps in different versions?). We identified 16 such libraries, and their proper coordinates.
  • Source artifacts
    Eclipse plug-ins have their source code in corresponding .source plug-ins. Maven has a similar convention, just using a “classifier” instead of appending to the artifact name. In Maven we conform to their convention, so that tools like m2e can correctly pick up the source code from any dependencies.
  • Other meta data
    Followed a hunt for project url, scm coordinates, artifact descriptions and related data. Much of this could be retrieved from our MANIFEST.MF files, some information is currently mapped using a static, manually maintained mapping. Other information like licences and organization are fully static during this process. In the end all was approved by the validation on OSSRH servers.

If you want to browse the resulting wealth, you may start at

Everything with fully qualified artifact names in these groups (and date of 2017-01-07 or newer) should be from the new, “official” upload.

This is just the beginning

The bug on which all this has been booked is Bug 484004: Start publishing Eclipse platform artifacts to Maven central. See the word “Start”?

To follow-up tasks are already on the board:

(1) Migrate all the various scripts, tools, and models to the proper git repo of our releng project. At the end of the day, this process of transformation and upload should become a routine operation to be invoked by our favourite build meisters.

(2) Fix any quirks in the generated pom files. E.g., we already know that the process did not handle fragments in an optimal way. As a result, consuming SWT from the new upload is not straight forward.

Both issues should be handled in or off bug 510072, in the hope, that when we publish Neon.3 the new, “official” Maven coordinates of Eclipse artifacts will be even fit all all real world use. So: please test and report in the bug any problems you might find.

(3) I was careful to say “Eclipse Project”. We don’t yet have the magic wand to apply this to literally all artifacts produced in the Eclipse community. Perhaps s.o. will volunteer to apply the approach to everything from the Simultaneous Release? If we can publish 300+ artifacts, we can also publish 7000+, can’t we? 🙂

happy building!

Written by Stephan Herrmann

January 9, 2017 at 23:21

Posted in Eclipse, Uncategorized

Tagged with , , , ,

How to kick the fly-shuttle

leave a comment »

This post was originally written on August 6th, 2013, but got lost when moving this blog. Luckily I could recover it from backup.

Software composition by “weaving”

Programming languages in the wider field of Aspect Oriented Software Development introduce some kinds of composition concepts, that cannot directly be mapped to pure Java. We got used to speaking of “weaving” as a mechanism for preparing classes in these languages for execution on the JVM, whereas a true aspect oriented virtual machine would natively perform this composition transparently and more efficiently. But with the dominance of the JVM it will be “weaving” for a long time to come.

fly-shuttle

Back in the early days, AOSD languages performed some static composition on source or class files. When OT/J entered the stage, first frameworks emerged that allowed us to hook into the class loading process, so we could do the weaving dynamically on-the-fly, without ever storing the woven classes on disk. Work is (still) in the pipe-line to support even runtime-(re)weaving.

One theme in all the development of Object Teams is: first solve the dynamic case, that’s where the challenges lie, and leave more static cases for later as an optimization. “Later” has come now.

How (not) to hook into class loading?

A little while into the development of OT/J, Java 5 was published introducing the JPLIS API and the concept of a -javaagent. Today many instrumentation tools use this facility and OT/J was among the first on this boat. So for standalone Java applications the problem is solved.

Later, while developing Eclipse plug-ins, we noticed that it wouldn’t be overly cool if you have to invoke Eclipse with a -javaagent. Not only is this very inconvenient for installation, it is also problematic (e.g., for performance) to feed all classes of Eclipse through our weaver. We were lucky again, and the Equinox Adaptor Hooks emerged right when we needed them. With these hooks we were able to integrate OT/J in a way that enables all the on-the-fly flexibility we want while at the same time it lets us control the process so that the flexibility cannot be abused. So in Equinox (or better: OT/Equinox) the problem is (more than) solved.

The fact that Eclipse Luna will no longer support these hooks was slightly alarming news for Object Teams, but that problem is already essentially solved – details to be reported soon.

I’ve heard about successful experiments running OT/J code in tomcat, but, hey, the list of application servers/containers is long, and if we need specifically crafted integration for each environment, it’s kind of difficult to argue that OT/J runs “everywhere”. Is there a generic why to kick the fly-shuttle?

fly-shuttle-2

Dynamic weaving is cool, but not having a fallback is uncool, so I finally developed the static scenario as a special case of the dynamic one: a build-time weaver for OT/J.

The little new tool is ridiculously simple: we have a weaver capable of load-time weaving. All I had to do is: create a little utility that would invoke this weaver ahead-of-time, i.e., during building.

While I’m not a big fan of Maven, in my day job Maven is a must, so I bit the bullet used the opportunity to learn a bit about development of Maven plug-ins. I was amazed to find lots of tutorials still suggesting to use API that are deprecated in Maven 3 and more surprised to read forum threads suggesting to still use that deprecated API, partly to be compatible with Maven 2 (OK?) and partly because deprecation doesn’t always mean there is an alternative (OOPS?). It’s also interesting to see, that much fundamental, up-to-date information is not available from a central source but only from tribal knowledge in hundreds of forums, blogs and whatnots. — Enough whining, on to the new stuff.

For simple projects, all you need now is a pom that looks about like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 
    <modelVersion>4.0.0</modelVersion>
 
    <parent>
        <groupId>org.eclipse.objectteams</groupId>
        <artifactId>objectteams-parent-pom</artifactId>
        <version>2.2.0</version>
    </parent>
 
    <artifactId>OTStopwatch_Built-Time_Weaver_Example</artifactId>
    <version>0.0.1-SNAPSHOT</version>
 
    <properties>
        <tycho.version>0.18.0</tycho.version>
        <otj.version>2.2.0</otj.version>
    </properties>
 
    <build>
        <plugins>
            <plugin>
                <groupId>org.eclipse.objectteams</groupId>
                <artifactId>objectteams-weaver-maven-plugin</artifactId>
                <version>0.8.0-SNAPSHOT</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>weave</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <teamClasses>
                        <teamClass>org.eclipse.objectteams.example.stopwatch.WatchUI</teamClass>
                        <teamClass>org.eclipse.objectteams.example.stopwatch.WatchUIAnalog</teamClass>
                    </teamClasses>
                </configuration>
            </plugin>
        </plugins>
    </build>
 
    <repositories>
        <repository>
            <id>ObjectTeamsRepository</id>
            <name>Object Teams Repository</name>
            <url>http://download.eclipse.org/objectteams/maven/3/repository</url>
        </repository>
    </repositories>
 
   <dependencies>
       <dependency>
           <groupId>org.eclipse.objectteams</groupId>
           <artifactId>objectteams-runtime</artifactId>
           <version>${otj.version}</version>
       </dependency>
   </dependencies>
</project>
 

Here we enable OT/J compilation via the parent pom, specify a repo where to get the OT artifacts, a dependency and a few versions, nothing sophisticated.

What’s new is the objectteams-weaver-maven-plugin, which is minimally configured just by specifying a list of team classes that should be woven. Oops, that’s already it.

By integrating this into your build, you’ll produce class files that can be directly executed “everywhere”, i.e., without the OT/J load-time weaver, and thus: without fiddling with the class loading process. One caveat remains: naturally, by producing the woven class files offline, it will be your responsibility to correctly feed the correct versions of class files into the classpath in the desired order. If you can’t use the load-time weaver, it won’t be able to help keep things simple.

More configuration options are documented on the plug-in’s site and more explanation using our favorite examples can be found in the wiki.

Here you have it: static weaving falls of naturally if you already have dynamic weaving. And the resulting class files can truly be run “everywhere”.

I’d be curious to hear of the first OT/J apps on Android (though personally I’d prefer ubuntu).

Written by Stephan Herrmann

January 28, 2014 at 18:16

Posted in Eclipse, Object Teams, OTEquinox

Tagged with , , ,

Compiling OT/Equinox projects using Tycho

with 3 comments

In a previous post I showed how the tycho-compiler-jdt Maven plug-in can be used for compiling OT/J code with Maven.

Recently, I was asked how the same can be done for OT/Equinox projects. Given that we were already using parts from Tycho, this shouldn’t be so difficult, right?

Once you know the solution, finding the solution is indeed easy, also in this case. Here it is:

We use almost the same declaration as for plain OT/J applications:

    <pluginManagement>
	<plugin>
	    <groupId>org.eclipse.tycho</groupId>
	    <artifactId>tycho-compiler-plugin</artifactId>
	    <version>${tycho.version}</version>
	    <extensions>true</extensions>
	    <dependencies>
		<dependency>
		    <groupId>org.eclipse.tycho</groupId>
		    <artifactId>tycho-compiler-jdt</artifactId>
		    <version>${tycho.version}</version>
		    <exclusions>
			<!-- Exclude the original JDT/Core to be replaced by the OT/J variant: -->
			<exclusion>
			    <groupId>org.eclipse.tycho</groupId>
			    <artifactId>org.eclipse.jdt.core</artifactId>
			</exclusion>
		    </exclusions>
		</dependency>
		<dependency>
		    <!-- plug the OT/J compiler into the tycho-compiler-jdt plug-in: -->
		    <groupId>org.eclipse</groupId>
		    <artifactId>objectteams-otj-compiler</artifactId>
		    <version>${otj.version}</version>
		</dependency>
	    </dependencies>
	</plugin>
    </pluginManagement>

So, what’s the difference? In both cases we need to adapt the tycho-compiler-jdt plug-in because that’s where we replace the normal JDT compiler with the OT/J variant. However, for plain OT/J applications tycho-compiler-jdt is pulled in as a dependency of maven-compiler-plugin and must be adapted on this path of dependencies, whereas in Tycho projects tycho-compiler-jdt is pulled in from tycho-compiler-plugin. Apparently, the exclusion mechanism is sensitive to how exactly a plug-in is pulled into the build. Interesting.

Once I figured this out, I created and published a new version of our Maven support for Object Teams: objectteams-parent-pom:2.1.1 — publishing Maven support for Object Teams 2.1.1 was overdue anyway 🙂

With the updated parent pom, a full OT/Equinox hello world pom now looks like this:

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
 
    <parent>
        <!-- We use Object Teams: -->
        <groupId>org.eclipse</groupId>
        <artifactId>objectteams-parent-pom</artifactId>
        <version>2.1.1</version>
    </parent>
 
    <artifactId>OTEquinox-over-tycho</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <!-- We are building an Eclipse plug-in (or OSGi bundle): -->
    <packaging>eclipse-plugin</packaging>
 
    <repositories>
 
        <!-- This is where we get all Object Teams stuff from: -->
        <repository>
            <id>ObjectTeamsRepository</id>
            <name>Object Teams Repository</name>
            <url>http://download.eclipse.org/objectteams/maven/3/repository</url>
        </repository>
 
        <!-- Add any p2 repositories needed for your application, e.g.: -->
        <repository>
            <id>Juno</id>
            <name>Eclipse Juno Repository</name>
            <url>http://download.eclipse.org/releases/juno</url>
            <layout>p2</layout>
        </repository>
    </repositories>
 
    <build>
        <plugins>
            <plugin>
                <!-- We build this project using Tycho: -->
                <groupId>org.eclipse.tycho</groupId>
                <artifactId>tycho-maven-plugin</artifactId>
                <extensions>true</extensions>
            </plugin>
        </plugins>
    </build>
</project>

Looks pretty straight forward, right?

To see the full OT/Equinox Hello World Example configured for Maven/Tycho simply import OTEquiTycho.zip as a project into your workspace.

cheers,
Stephan

Written by Stephan Herrmann

October 31, 2012 at 00:16

Builds are like real software – or even more so

with 8 comments

Being a part-time release engineer for the Object Teams project I can only agree with every word Kim writes about the job, I wish I could hire her for our project 🙂

She writes:

“Nobody in needs to understand how the build works, they just need to push a button. That’s great. Until the day before a release when your build fails with a cryptic message about unresolved dependencies. And you have no idea how to fix it. And neither does anyone else on the team.”

That puts a sad smile on my face and I’d like to add a little quality metric that seems cruel for today’s build systems, but might actually be useful for any software:

No software can be better than its worst error message.

One extreme I experienced was in a PDE/Build-ant-build which I had to set to verbose to get any useful answer but then I had to find the relevant error message deeply buried in literally tens of megabytes of log output. Takes ages to browse that log file. Other tools rank towards the other end of the spectrum saying basically “it didn’t work”.

Why is the worst error message relevant? When you hit that worst message it’s close to saying “game over”. Especially when working on a build I’ve come to the point time and again where all my creativity and productivity came to a grinding halt and for days or weeks I simply made zero progress because I had no idea why that system didn’t work and what it expected me to do to fix the thing. Knock-out.

Obviously I hate that state when I make no progress towards my goal. And typically that state is reached by poor communication from some framework back to me.

Real coolness

I know people usually don’t like to work on improving error messages, but please, don’t think good error messages are any bit less cool than running your software on mars. On the one hand we try to build tools that improve developers’ productivity by a few percent and than the tool will give answers that bring that very productivity down to zero. That’s – inconsistent.

I’m tempted to repeat the p2 story here. Many will remember the merciless dump of data from the sat solver that p2 gave in its early days. Some will consider the problem solved by now. Judge for yourself: what’s the worst-case time a regular Eclipse user will need to understand what p2 is telling him/her by one of its error messages.

The intention of this post is not to blame any particular technology. The list would be quite long anyway. It’s about general awareness (big words, sorry 🙂 ).

Consider the worst case

Again, why worst case? Because the worst case will happen. And it’s enough if it hits you once to easily compensate all the time savings the tool otherwise brought to you.

Communicate!

Framework developers, tool smiths: let your software communicate with the user and let it be especially helpful when the user is in dire need of help.

One small contribution in this field I’d like to share with you: in the OTDT every error/warning raised by the compiler not only tries to precisely describe what’s wrong but it is directly linked to the corresponding paragraph in the language definition that is violated by the current code. At least this should completely explain why the current code is wrong. It’s a small step, but I feel a strong need for linking specific help to every error message.

But first, the software has to anticipate every single error that will occur in order to produce useful messages. That’s the real reason why creating complex software is so challenging. Be it a build system or the “real” software.

Be cool, give superb error messages!

Written by Stephan Herrmann

September 4, 2011 at 15:28

Posted in Eclipse, OTDT

Tagged with , , ,

Between the Times

with 4 comments

This week is (supposed to be) “quiet week” at Eclipse: Release Candidate 4 is in place and more testing is being done before the final Indigo Release. Time to speak of s.t. that the Object Teams project has neglected a bit in the past: compiling / building with Maven.

Edit (June 2014): In a good tradition of using quiet week I’ve updated the Maven support for OT/J 2.3.0 (Luna).

Compiling OT/J with Maven

In the days of Maven2, pluging-in a non-standard compiler required fiddling with the nexus compiler plugin, which was buggy and all that. With the recent development around Maven3 and Tycho things became easier. The following pom-snippet is all you need to tell Maven to use the OT/J compiler:

<build>
    <pluginManagement>
        <plugins>
            <plugin>
                <!-- Use compiler plugin with tycho as the adapter to the OT/J compiler. -->
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                    <compilerId>jdt</compilerId>
                </configuration>
                <dependencies>
                    <!-- This dependency provides the implementation of compiler "jdt": -->
                    <dependency>
                        <groupId>org.sonatype.tycho</groupId>
                        <artifactId>tycho-compiler-jdt</artifactId>
                        <version>${tycho.version}</version>
                        <exclusions>
                            <!-- Exclude the original JDT/Core to be replaced by the OT/J variant: -->
                            <exclusion>
                                <groupId>org.sonatype.tycho</groupId>
                                <artifactId>org.eclipse.jdt.core</artifactId>
                            </exclusion>
                        </exclusions>
                    </dependency>
                    <dependency>
                        <!-- plug the OT/J compiler into the tycho-compiler-jdt plug-in: -->
                        <groupId>org.eclipse</groupId>
                        <artifactId>objectteams-otj-compiler</artifactId>
                        <version>${otj.version}</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </pluginManagement>
</build>
 
Edit (June 2014): As of version 2.3.0 (Luna) the groupId has been changed to org.eclipse.objectteams (meanwhile, 3-part group IDs are the de-facto standard for Eclipse projects).

This relies on the fact, that the OT/J compiler is compatible with the JDT compiler, good.

To make things go smooth in various build phases a few more bits are required:

  • provide the OT/J runtime jar and its dependency (bcel)
  • tell surefire about required command line arguments for running OT/J programs

These basics are collected in a parent pom so that all you need are these two snippets:

<repositories>
    <repository>
        <id>ObjectTeamsRepository</id>
        <id>ObjectTeamsRepository</id>
        <name>Object Teams Repository</name>
        <url>http://download.eclipse.org/objectteams/maven/3/repository</url>
    </repository>
</repositories>
<parent>
    <groupId>org.eclipse</groupId>
    <artifactId>objectteams-parent-pom</artifactId>
    <version>2.0.0</version>
</parent>
 
Edit (June 2014): Again: as of version 2.3.0 the groupId is org.eclipse.objectteams.

This converts any Java project into an OT/J project, simply enough, huh?

At least compiling and testing works out-of-the box. Let me know what additional sophistication you need and if adjustments to other build phases are needed.

OT/J over m2e

Well, I’m not much of a Maven expert, but I’m an Eclipse enthusiast, so, how can we use the above inside the IDE?

The bad news: the m2e plug-in has hardcoded its support for the compiler plugin to use “javac” as the compiler. Naive attempts to use tycho-compiler-jdt have failed with the infamous "Plugin execution not covered by lifecycle configuration".

The good news: By writing a tiny OT/Equinox plug-in I was able to remedy those issues. If you have the m2e plug-in installed (Indigo version), the normal “Update Project Configuration” action will suffice:
Update Project Configuration

After that, the project is recognized as an OT/J project, both by the IDE …

… and also by maven …
Running mvn test

Where to get it?

The mentioned plug-in is brand-new and as such it is not part of the regular OTDT distribution. OTOH, the plug-in is so simple, I don’t expect a lot of changes needed. So, if you have an Indigo-ish Eclipse just add this update site:

Then install “Maven Integration for Object Teams (Early Access)”:
Install the Maven Integration for Object Teams

Don’t forget the <repository> and <parent> definitions from above and you should be ready to go! Yesss, sometimes it pays off that OT/J is a kind of Java, a lot of the tooling can just piggy-back on what’s already there for Java 🙂

Written by Stephan Herrmann

June 14, 2011 at 14:56

Posted in Eclipse, Object Teams

Tagged with , , ,

Moving business

leave a comment »


Remember the last time you had to cram your whole hosehold into boxes, bags and cases? You may feel excited about your new home etc. but the whole boxing business is quite a drag, ain’t it? There’s of course at least two ways of approaching this:

  1. Don’t look, just shovel everything randomly into boxes
  2. Look at each single piece, indulge in memories associates with it and sort it to its likes

Obviously, (1) means that on the other side you will unpack a whole mess of junk. OTOH, (2) won’t be finished before the moving truck arives. Still deep inside there lives a bit of hope, that you’ll move to your new home with only that stuff you’ll actually want, and everything ready to be neatly deployed into its new destination. Moving could free you from all the junk you don’t want any more, right? And even more, after the move, you may want to know, where everything is, right?

When moving the Object Teams project to Eclipse I was in the lucky situation that I could indeed use the occasion to sort through some of our stuff. The software engineer might be tempted to even speak of some “quality assurance” along the way, but let’s be careful with our wording for now.

On January 26 this year, the Eclipse Object Teams Project was created and we started to put up signs “Object Teams is moving to Eclipse.org”. Recently, I changed that sign to “Object Teams has moved to Eclipse.org”. So, what exactly happened between then and now?

Learning the infrastructure

At first the newly appointed eclipse committer and project lead is overwhelmed with all the shiny technology: web server, wiki, version repositories, build servers, download servers, the portal, project metadata, accounts for this and that, bugzilla components, versions and milestones and what-not. “Alles so schön bunt hier!”

I won’t indulge in talking about the paper work needed at this stage, after some four days I had most my accounts and the project was registered as “incubation – conforming”, so we were ready to go into the parallel IP process.

Initial contribution

On project day #12 I submitted my first CQ and, yes, that submission was already the result of heavy refactoring: I had renamed most (not all!!) occurrences of org.objectteams to org.eclipse.objectteams. A piece of cake? Well not exactly if the code piles up to a zip of 35 MBytes (not including .class files and nested archives), and not if your team-support plug-in goes berserk on some of the renamings and if … (see also this post).

Parallel IP process – our version

In fact our version of the parallel IP process looked like this:

On one thread I was chasing after some people and institutions to just provide the necessary signed statements of code provenance. Wrt the individuals this was painless, however, the university and the research institute involved both had their very specific strategies for delaying the project. All-in-all it took them more than one week per sentence in the final document. Or would you prefer the words-per-day count?   The much feared IP analysis turned out to be a very constructive collaboration with Sharon Corbett. I was really amazed about the obscure pieces of code (and comments) she brought to light, things that I never knew where in there. So that was helpful information, actually 🙂 .
Most of all I was pleased by her quick responsiveness – quite in contrast to the other thread. Thanks Sharon!
I should also thank Jan Wloka who from the outset of the project took care that we’d have copyright headers and that stuff.

The effect was: at the time I got the signatures that cleared us to check our sources into svn the IP analysis was already done and complete! And not only that, during that process we’ve done some significant cleanup.

Code cleanup triggered by the move:

  • Proviously, Object Teams used the JMangler framework for launching with our bytecode transformers in place. This was a great thing to have back in the olden Java 1.3 times. But in 2010 our Java 5 based alternative had matured and we didn’t even have to put JMangler into any of our moving boxes 🙂
  • We used to maintain a patched version of BCEL 4.4.1 (developed at Freie Universität Berlin, as the namespace de.fub still announced). I consulted the AspectJ folks who maintain a patched version, too. But their patch only vaguely resembles the original, and they clearly stated that they saw now chances of these changes being adopted upstream. So, I went back to our sources, checked the patches, checked the current version 5.2 of bcel and found that the remaining bugs could actually be easily worked around from the outside. That’s when I learned the details of Orbit: since our initial commits to Eclipse we never had to bother about the bcel version, it just comes right flying from the Orbit. So we got that legacy version cleared up.
  • My heart skipped a tiny single beat, when I learned that one of our most central data structures could not be accepted at Eclipse: I had patched class WeakHashMap from the OpenJDK to create a DoublyWeakHashMap with quite unique characteristics concerning garbage collection. We need that! Yet, the license (GPL with “classpath exception”) was not accepted. I made a quick experiment with wrapping instead of patching and guess what I learned (again): while the patched version was created in the pursuit of performance, still, after changing the strategy (to what was destined to be slower) my measurements could not show any performance penalty. So, carve that in stone: never optimize without measuring. The new version has the same performance – and no license issues!

Of course, there were more issues like needing to file a new CQ just for using files like xhtml1-strict.dtd, but those caused no grief after all. Enter the next phase.

Getting everything to build and test on eclipse servers

OK, when we opened the boxes at our new home, some of the content was actually broken on the way.

Fixing broken builds

The ugliest part was getting an ancient set of PDE/Build scripts to run on build.eclipse.org. Digging through a 30MByte build-log looking for the cause of a build failure never was fun. The point that dissatisfies me with all the build technology I’ve seen so far: You have a build that works on one machine and with one version of the software. Then, one arbitrary piece of the setup changes, let’s take a big change as the example: moving your JDK from 5 to 6. It’s OK that things may break at this point, but the kind of breakage frequently seems to have nothing to do with what you’ve changed, e.g., after moving to a different JDK the compiler can no longer resolve java.long.Object, whereas everybody knows: that’s not the difference between the two JDKs. The problem is not broken builds, the problem is how little clues the logs give you for finding the root cause of the breakage, or even: telling you how to fix it. A technology that works when it works is one thing. A technology that helps you get it to work is another (and we’re working hard to make Object Teams fall into the second category).

Modernizing the build

OK, enough complained, the move to Eclipse again gave reason to cleanup that monster build and even update to using some of the automatic built-in p2 stuff (yes, finally we use p2.gathering=true), rather than manually invoking the various p2 applications (publisher, director). When it runs, you may even get the impression you know why it does.

The final round in improving our build was adding bundle signing, yeah! Of course, that’s when all the p2-metadata generated during the build don’t help you any more because those include the checksums of your unsigned jars. So I created a tiny little shell script to automate those steps required after a successful build&test. I ended up with 7 more transformations of our metadata needed at this stage. So we’re back at directly invoking p2 publisher, p2 director, do various XSL transformations etc. Most of these could actually be done by a PDE/Build – p2 integration, but let’s not expect too much, not now.

Did I mention the almost 50000 tests that successfully run during each build? Well, that’s what we owe our users, right?

It builds – let’s ship it!

OK, let’s move ahead to the success-story-part. Less than a month after the initial commit we had our first milestone. It’s so good to be back in business 🙂 After fixing all those migration-induced regressions I’m sure our code has better quality than before.

User side migration

Only one burden we had to pass on to users: due to the changed namespace, the configuration files of existing OT/J projects have to be updated. Luckily, it wasn’t too difficult to add some specific build-path problems and quickfixes, which should make the migration for users pretty smooth.

Installing

Right while I was publishing our first milestones a new cool tool came around the corner: the Market Place Client. So, now if you download, e.g., the Helios Package “Eclipse IDE for Java Developers” you’ll get the OTDT installed without having to know the download address, just select Help > Eclipse Marketplace and search for “Object Teams”, and you’re ready to hit “Install“:

Interestingly, in order for this to work I had to ask for this feature, which later down the road triggered this blocker security issue. At the end of the day this made me ponder about generalizing various things that the user might want to know when installing software. And indeed, Object Teams should play an active role in this discussion: the whole business of OT/Equinox is based on the assumption that the user agrees with what we are doing. We already do our best in treating the users as grown-ups who can make their own decisions, if we provide sufficient
information, like:

This little screenshot tells you a whole story about this version of the bundle org.eclipse.jdt.core (see last column):

  • The icon in the 1st column says: this bundle is signed, but that signed content is going to be woven at load time before the JVM sees it
  • Columns 2 & 3 give the obvious information that this is not the version provided by the JDT team but something from the Object Teams project, which BTW. is still in its incubation phase.
  • Column 4 finally gives you all the gory details: a sophisticated version number plus the list of OT/Equinox plugins that have declared to adapt the current plugin.

That’s the kind of transparency we show upon request after the software is installed. The mentioned bug 316702 is about providing similar transparency already during install.

So, what’s the plan?

Given that all legal and technical matters have been sorted out to this point, and given that the tool is in an even better shape than the final OTDT 1.4.0 from objectteams.org, what’s our plan?

  • Just recently I requested a Release Review, tentatively scheduled for July 7, so with only little delay after Helios we should have an actual, stable release.
  • I decided to defer the project graduation some more months to give us time to define which parts of the software are actually API.

Where can I see it in action?

Well, given the current milestone releases (and the ease of installing) and all the documentation we have in the wiki nothing should stop you from running your first Object Teams demo in do-it-yourself mode 🙂

Otherwise, if you happen to be in Vienna on June 25, just come to this DemoCamp and I’ll help you to get started with Object Teams.

So, indeed for the Object Teams project that past 6 months were used to turn this:

into this:

Written by Stephan Herrmann

June 15, 2010 at 23:06

Posted in Eclipse, Object Teams, OTDT, OTEquinox

Tagged with , , ,