Eclipse Neon.2 is on Maven Central

It’s done, finally!

Bidding farewell to my pet peeve

In my job at GK Software I have the pleasure of developing technology based on Eclipse. But those colleagues consuming my technology work on software that has no direct connection to Eclipse nor OSGi. Their build technology of choice is Maven (without tycho that is). So whenever their build touches my technology we are facing a “challenge”. It doesn’t make a big difference if they are just invoking a code generator built using Xtext etc or whether some Eclipse technology should actually be included in their application runtime.

Among many troubles, I recall one situation that really opened my eyes: one particular build had been running successfully for some time, until one day it was fubar. One Eclipse artifact could no longer be resolved. Followed long nights of searching why that artifact may have disappeared, but we reassured ourselves, nothing had disappeared. Quite to the contrary somewhere on the wide internet (Maven Central to be precise) a new artifact had appeared. So what? Well, that artifact was the same that we also had on our internal servers. Well, if it’s the same, what’s the buzz? It turned out it had a one-char difference in its version: instead of 1.2.3.v20140815 its version was 1.2.3-v20140815. Yes take a close look, there is a difference. Bottom line, with both almost-identical versions available, Maven couldn’t figure out what to do, maybe each was considered as worse than the other, to the effect that Maven simply failed to use either. Go figure.

More stories like this and I realized that relying on Eclipse artifacts in Maven builds was always at the mercy of some volunteers, who typically don’t have a long-term relationship to Eclipse, who filled in a major gap by uploading individual Eclipse artifacts to Maven Central (thanks to you volunteers, please don’t take it personally: I’m happy that your work is no longer needed). Anybody who has ever studied the differences between Maven and OSGi (wrt dependencies and building that is) will immediately see that there are many possible ways to represent Eclipse artifacts (OSGi bundles) in a Maven pom. The resulting “diversity” was one of my pet peeves in my job.

At this point I decided to be the next volunteer who would screw up other people’s builds who would collaborate with the powers that be at Eclipse.org to produce the official uploads to Maven Central.

As of today, I can report that this dream has become reality, all relevant artifacts of Neon.2 that are produced by the Eclipse Project, are now “officially” available from Maven Central.

Bridging between universes

I should like to report some details of how our artifacts are mapped into the Maven world:

The main tool in this endeavour is the CBI aggregator, a model based tool for transforming p2 repositories in various ways. One of its capabilities is to create a Maven repository (a dual use repo actually, but the p2 side of this is immaterial to this story). That tool does a great job of extracting meta data from the p2 repo in order to create “meaningful” pom files, the key feature being: it copies all dependency information, which is originally authored in MANIFEST.MF, into corresponding declarations in the pom file.

Still a few things had to be settled, either by improving the tool, by fine tuning the input to the tool, or by some steps of post-processing the resulting Maven repo.

• Group IDs
  While OSGi artifacts only have a single qualified Bundle-SymbolicName, Maven requires a two-part name: groupId x artifactId. It was easy to agree on using the full symbolic name for the artifactId, but what should the groups be? We settled on these three groups for the Eclipse Project:
  • org.eclipse.platform
  • org.eclipse.jdt
  • org.eclipse.pde
• Version numbers
  In Maven land, release versions have three segments, in OSGi we maintain a forth segment (qualifier) also for releases. To play by Maven rules, we decided to use three-part versions for our uploads to Maven Central. This emphasizes the strategy to only publish releases, for which the first three parts of the version are required to be unique.
• 3rd party dependencies
  All non-Eclipse artifacts that we depend on should be referenced by their proper coordinates in Maven land. By default, the CBI aggregator assigns all artifacts to the synthetic group p2.osgi.bundle, but if s.o. depends on p2.osgi.bundle:org.junit this doesn’t make much sense. In particular, it must be avoided that projects consuming Eclipse artifacts will get the same 3rd party library under two different names (perhaps in different versions?). We identified 16 such libraries, and their proper coordinates.
• Source artifacts
  Eclipse plug-ins have their source code in corresponding .source plug-ins. Maven has a similar convention, just using a “classifier” instead of appending to the artifact name. In Maven we conform to their convention, so that tools like m2e can correctly pick up the source code from any dependencies.
• Other meta data
  Followed a hunt for project url, scm coordinates, artifact descriptions and related data. Much of this could be retrieved from our MANIFEST.MF files, some information is currently mapped using a static, manually maintained mapping. Other information like licences and organization are fully static during this process. In the end all was approved by the validation on OSSRH servers.

If you want to browse the resulting wealth, you may start at

• https://repo1.maven.org/maven2/org/eclipse/platform/
• https://repo1.maven.org/maven2/org/eclipse/jdt/
• https://repo1.maven.org/maven2/org/eclipse/pde/

Everything with fully qualified artifact names in these groups (and date of 2017-01-07 or newer) should be from the new, “official” upload.

This is just the beginning

The bug on which all this has been booked is Bug 484004: Start publishing Eclipse platform artifacts to Maven central. See the word “Start”?

To follow-up tasks are already on the board:

(1) Migrate all the various scripts, tools, and models to the proper git repo of our releng project. At the end of the day, this process of transformation and upload should become a routine operation to be invoked by our favourite build meisters.

(2) Fix any quirks in the generated pom files. E.g., we already know that the process did not handle fragments in an optimal way. As a result, consuming SWT from the new upload is not straight forward.

Both issues should be handled in or off bug 510072, in the hope, that when we publish Neon.3 the new, “official” Maven coordinates of Eclipse artifacts will be even fit all all real world use. So: please test and report in the bug any problems you might find.

(3) I was careful to say “Eclipse Project”. We don’t yet have the magic wand to apply this to literally all artifacts produced in the Eclipse community. Perhaps s.o. will volunteer to apply the approach to everything from the Simultaneous Release? If we can publish 300+ artifacts, we can also publish 7000+, can’t we? 🙂

happy building!

New Honours, New Synergy

I’m flattered to see my name appear on one more Eclipse web-page:

Thanks to the JDT/Core team for accepting me as a new member!

Actually, I’ve been poking about the JDT source code since 2003 when we started implementing the Object Teams Development Tooling (OTDT) on top of the JDT. So it was only natural that I would stumble upon a bug in the JDT every once in a while (incl. one “greatbug” 🙂 ). In this specific situation, reporting bugs and trying to help find solutions was more than just good Eclipse citizenship: the discussions in bugzilla also helped me to better understand the JDT sources and thus it helped me developing the OTDT. And that’s what this post is about: synergy!

Personal Goals

Actually, watching the JDT/Core bug inbox provides a constant stream of cool riddles: spooky Java examples producing unexpected compiler/runtime results. Sometimes those are real fun to crack. Which compiler is right, which one isn’t? What’s the programmer trying to do? I’d say there’s no better way to test your Java knowledge 🙂

Also, watching this stream helps me stay informed, because every patch will eventually need to be merged into the Object Teams branch of the JDT/Core.

But I also have some pet projects that I’d like to actively push forward. Currently I’m thinking about these two:

• Supporting inter-procedural null analysis
• Making the compiler closer match the OSGi semantics

Inter-procedural null analysis

Naturally, the JDT/Core is a facilitator for tons of cool functionality in the IDE, but it hardly communicates directly with the user. But we have one channel, where the compiler can actually talk to the developer and give advice that’s well worth your bucks: errors and warnings. Did you know, that the compiler can immediately tell you what’s wrong about this piece of code:

void foo(String in) {
   if (in == null) new IllegalArgumentException("Null is not allowed here");
   ... // real work done here
 

(Yep, that was bug 236385, see the New&Noteworthy on how to enable).

Many Java developers share the experience that one of the most frequent problems is also one of the most mundane: NPE. Those, who are aware of this easily produce the opposite problem: cluttering there code with meaningless null-checks (many of which have no better action than the above IllegalArgumentException which isn’t much better than throwing the NPE in the first place). The point is, we’d want to know which values can actually be null (and why!) so we write only the necessary null-checks and for those we should think really hard what would be a suitable action, right? (“This shouldn’t happen” is not an answer!).

Thus I’m happy I could get involved in some improvements of the compiler’s flow analysis. I also experienced that null-analysis combined with aggressive optimization is a delicate issue – remember SDK 3.7M2a? Sorry about that, but to my justification I might add that bug 325755 has always been dormantly present, I only woke’em up. I must admit for those 105 minutes my heart beat was a bit above average 😉 .

Anyway, the current null analysis is still fundamentally limited: it knows nothing about nullness of method arguments nor method call results. So it would be a real cool enhancement if we could feed such information into the compiler. Fortunately, the theory of how to do this (you may either think of improved type systems or of design by contract) is a well-explored subject. Actually, bug 186342 already has some patches in this direction. Since discussing in bugs with long history is a bit tedious I created this wiki page. Sadly, it’s not a technical difficulty that’s keeping us from immediately releasing that patch, but a stalled standardization process 😦 .

However, as outlined in the wiki, here comes another synergy: in case we can’t find a solution that will be sufficiently compatible with future standards, I can easily create an early-adopters release, so sorting out the details of implementation and usage can be done in parallel with the standardization process. How that? I’d simply ship the compiler enhancement as an OT/Equinox-enabled plug-in. This would allow us to ship the compiler enhancement as separate plug-in ready to be tried by early adopters.

OSGi-aware compiler

Some of you may have noticed (here, or here, or here, or here or one of the many duplicates) that the compiler’s concept of a (single, linear) classpath is not a good match for compiling OSGi code. I will soon write another wiki page with my current thinkings about that issue. Stay tuned …

Comments?

For both issues I’d appreciate any comments / feedback. I’ll be checking updates of the bugs I listed, or the wiki page or … talk to me during ESE! Let’s move something together.

See you all in Ludwigsburg!
Stephan