The Object Teams Blog

Adding team spirit to your objects.

Posts Tagged ‘maven

Eclipse Neon.2 is on Maven Central

with 6 comments

It’s done, finally!

Bidding farewell to my pet peeve

In my job at GK Software I have the pleasure of developing technology based on Eclipse. But those colleagues consuming my technology work on software that has no direct connection to Eclipse nor OSGi. Their build technology of choice is Maven (without tycho that is). So whenever their build touches my technology we are facing a “challenge”. It doesn’t make a big difference if they are just invoking a code generator built using Xtext etc or whether some Eclipse technology should actually be included in their application runtime.

Among many troubles, I recall one situation that really opened my eyes: one particular build had been running successfully for some time, until one day it was fubar. One Eclipse artifact could no longer be resolved. Followed long nights of searching why that artifact may have disappeared, but we reassured ourselves, nothing had disappeared. Quite to the contrary somewhere on the wide internet (Maven Central to be precise) a new artifact had appeared. So what? Well, that artifact was the same that we also had on our internal servers. Well, if it’s the same, what’s the buzz? It turned out it had a one-char difference in its version: instead of 1.2.3.v20140815 its version was 1.2.3-v20140815. Yes take a close look, there is a difference. Bottom line, with both almost-identical versions available, Maven couldn’t figure out what to do, maybe each was considered as worse than the other, to the effect that Maven simply failed to use either. Go figure.

More stories like this and I realized that relying on Eclipse artifacts in Maven builds was always at the mercy of some volunteers, who typically don’t have a long-term relationship to Eclipse, who filled in a major gap by uploading individual Eclipse artifacts to Maven Central (thanks to you volunteers, please don’t take it personally: I’m happy that your work is no longer needed). Anybody who has ever studied the differences between Maven and OSGi (wrt dependencies and building that is) will immediately see that there are many possible ways to represent Eclipse artifacts (OSGi bundles) in a Maven pom. The resulting “diversity” was one of my pet peeves in my job.

At this point I decided to be the next volunteer who would screw up other people’s builds who would collaborate with the powers that be at Eclipse.org to produce the official uploads to Maven Central.

As of today, I can report that this dream has become reality, all relevant artifacts of Neon.2 that are produced by the Eclipse Project, are now “officially” available from Maven Central.

Bridging between universes

I should like to report some details of how our artifacts are mapped into the Maven world:

The main tool in this endeavour is the CBI aggregator, a model based tool for transforming p2 repositories in various ways. One of its capabilities is to create a Maven repository (a dual use repo actually, but the p2 side of this is immaterial to this story). That tool does a great job of extracting meta data from the p2 repo in order to create “meaningful” pom files, the key feature being: it copies all dependency information, which is originally authored in MANIFEST.MF, into corresponding declarations in the pom file.

Still a few things had to be settled, either by improving the tool, by fine tuning the input to the tool, or by some steps of post-processing the resulting Maven repo.

  • Group IDs
    While OSGi artifacts only have a single qualified Bundle-SymbolicName, Maven requires a two-part name: groupId x artifactId. It was easy to agree on using the full symbolic name for the artifactId, but what should the groups be? We settled on these three groups for the Eclipse Project:

    • org.eclipse.platform
    • org.eclipse.jdt
    • org.eclipse.pde
  • Version numbers
    In Maven land, release versions have three segments, in OSGi we maintain a forth segment (qualifier) also for releases. To play by Maven rules, we decided to use three-part versions for our uploads to Maven Central. This emphasizes the strategy to only publish releases, for which the first three parts of the version are required to be unique.
  • 3rd party dependencies
    All non-Eclipse artifacts that we depend on should be referenced by their proper coordinates in Maven land. By default, the CBI aggregator assigns all artifacts to the synthetic group p2.osgi.bundle, but if s.o. depends on p2.osgi.bundle:org.junit this doesn’t make much sense. In particular, it must be avoided that projects consuming Eclipse artifacts will get the same 3rd party library under two different names (perhaps in different versions?). We identified 16 such libraries, and their proper coordinates.
  • Source artifacts
    Eclipse plug-ins have their source code in corresponding .source plug-ins. Maven has a similar convention, just using a “classifier” instead of appending to the artifact name. In Maven we conform to their convention, so that tools like m2e can correctly pick up the source code from any dependencies.
  • Other meta data
    Followed a hunt for project url, scm coordinates, artifact descriptions and related data. Much of this could be retrieved from our MANIFEST.MF files, some information is currently mapped using a static, manually maintained mapping. Other information like licences and organization are fully static during this process. In the end all was approved by the validation on OSSRH servers.

If you want to browse the resulting wealth, you may start at

Everything with fully qualified artifact names in these groups (and date of 2017-01-07 or newer) should be from the new, “official” upload.

This is just the beginning

The bug on which all this has been booked is Bug 484004: Start publishing Eclipse platform artifacts to Maven central. See the word “Start”?

To follow-up tasks are already on the board:

(1) Migrate all the various scripts, tools, and models to the proper git repo of our releng project. At the end of the day, this process of transformation and upload should become a routine operation to be invoked by our favourite build meisters.

(2) Fix any quirks in the generated pom files. E.g., we already know that the process did not handle fragments in an optimal way. As a result, consuming SWT from the new upload is not straight forward.

Both issues should be handled in or off bug 510072, in the hope, that when we publish Neon.3 the new, “official” Maven coordinates of Eclipse artifacts will be even fit all all real world use. So: please test and report in the bug any problems you might find.

(3) I was careful to say “Eclipse Project”. We don’t yet have the magic wand to apply this to literally all artifacts produced in the Eclipse community. Perhaps s.o. will volunteer to apply the approach to everything from the Simultaneous Release? If we can publish 300+ artifacts, we can also publish 7000+, can’t we? 🙂

happy building!

Advertisements

Written by Stephan Herrmann

January 9, 2017 at 23:21

Posted in Eclipse, Uncategorized

Tagged with , , , ,

How to kick the fly-shuttle

leave a comment »

This post was originally written on August 6th, 2013, but got lost when moving this blog. Luckily I could recover it from backup.

Software composition by “weaving”

Programming languages in the wider field of Aspect Oriented Software Development introduce some kinds of composition concepts, that cannot directly be mapped to pure Java. We got used to speaking of “weaving” as a mechanism for preparing classes in these languages for execution on the JVM, whereas a true aspect oriented virtual machine would natively perform this composition transparently and more efficiently. But with the dominance of the JVM it will be “weaving” for a long time to come.

fly-shuttle

Back in the early days, AOSD languages performed some static composition on source or class files. When OT/J entered the stage, first frameworks emerged that allowed us to hook into the class loading process, so we could do the weaving dynamically on-the-fly, without ever storing the woven classes on disk. Work is (still) in the pipe-line to support even runtime-(re)weaving.

One theme in all the development of Object Teams is: first solve the dynamic case, that’s where the challenges lie, and leave more static cases for later as an optimization. “Later” has come now.

How (not) to hook into class loading?

A little while into the development of OT/J, Java 5 was published introducing the JPLIS API and the concept of a -javaagent. Today many instrumentation tools use this facility and OT/J was among the first on this boat. So for standalone Java applications the problem is solved.

Later, while developing Eclipse plug-ins, we noticed that it wouldn’t be overly cool if you have to invoke Eclipse with a -javaagent. Not only is this very inconvenient for installation, it is also problematic (e.g., for performance) to feed all classes of Eclipse through our weaver. We were lucky again, and the Equinox Adaptor Hooks emerged right when we needed them. With these hooks we were able to integrate OT/J in a way that enables all the on-the-fly flexibility we want while at the same time it lets us control the process so that the flexibility cannot be abused. So in Equinox (or better: OT/Equinox) the problem is (more than) solved.

The fact that Eclipse Luna will no longer support these hooks was slightly alarming news for Object Teams, but that problem is already essentially solved – details to be reported soon.

I’ve heard about successful experiments running OT/J code in tomcat, but, hey, the list of application servers/containers is long, and if we need specifically crafted integration for each environment, it’s kind of difficult to argue that OT/J runs “everywhere”. Is there a generic why to kick the fly-shuttle?

fly-shuttle-2

Dynamic weaving is cool, but not having a fallback is uncool, so I finally developed the static scenario as a special case of the dynamic one: a build-time weaver for OT/J.

The little new tool is ridiculously simple: we have a weaver capable of load-time weaving. All I had to do is: create a little utility that would invoke this weaver ahead-of-time, i.e., during building.

While I’m not a big fan of Maven, in my day job Maven is a must, so I bit the bullet used the opportunity to learn a bit about development of Maven plug-ins. I was amazed to find lots of tutorials still suggesting to use API that are deprecated in Maven 3 and more surprised to read forum threads suggesting to still use that deprecated API, partly to be compatible with Maven 2 (OK?) and partly because deprecation doesn’t always mean there is an alternative (OOPS?). It’s also interesting to see, that much fundamental, up-to-date information is not available from a central source but only from tribal knowledge in hundreds of forums, blogs and whatnots. — Enough whining, on to the new stuff.

For simple projects, all you need now is a pom that looks about like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 
    <modelVersion>4.0.0</modelVersion>
 
    <parent>
        <groupId>org.eclipse.objectteams</groupId>
        <artifactId>objectteams-parent-pom</artifactId>
        <version>2.2.0</version>
    </parent>
 
    <artifactId>OTStopwatch_Built-Time_Weaver_Example</artifactId>
    <version>0.0.1-SNAPSHOT</version>
 
    <properties>
        <tycho.version>0.18.0</tycho.version>
        <otj.version>2.2.0</otj.version>
    </properties>
 
    <build>
        <plugins>
            <plugin>
                <groupId>org.eclipse.objectteams</groupId>
                <artifactId>objectteams-weaver-maven-plugin</artifactId>
                <version>0.8.0-SNAPSHOT</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>weave</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <teamClasses>
                        <teamClass>org.eclipse.objectteams.example.stopwatch.WatchUI</teamClass>
                        <teamClass>org.eclipse.objectteams.example.stopwatch.WatchUIAnalog</teamClass>
                    </teamClasses>
                </configuration>
            </plugin>
        </plugins>
    </build>
 
    <repositories>
        <repository>
            <id>ObjectTeamsRepository</id>
            <name>Object Teams Repository</name>
            <url>http://download.eclipse.org/objectteams/maven/3/repository</url>
        </repository>
    </repositories>
 
   <dependencies>
       <dependency>
           <groupId>org.eclipse.objectteams</groupId>
           <artifactId>objectteams-runtime</artifactId>
           <version>${otj.version}</version>
       </dependency>
   </dependencies>
</project>
 

Here we enable OT/J compilation via the parent pom, specify a repo where to get the OT artifacts, a dependency and a few versions, nothing sophisticated.

What’s new is the objectteams-weaver-maven-plugin, which is minimally configured just by specifying a list of team classes that should be woven. Oops, that’s already it.

By integrating this into your build, you’ll produce class files that can be directly executed “everywhere”, i.e., without the OT/J load-time weaver, and thus: without fiddling with the class loading process. One caveat remains: naturally, by producing the woven class files offline, it will be your responsibility to correctly feed the correct versions of class files into the classpath in the desired order. If you can’t use the load-time weaver, it won’t be able to help keep things simple.

More configuration options are documented on the plug-in’s site and more explanation using our favorite examples can be found in the wiki.

Here you have it: static weaving falls of naturally if you already have dynamic weaving. And the resulting class files can truly be run “everywhere”.

I’d be curious to hear of the first OT/J apps on Android (though personally I’d prefer ubuntu).

Written by Stephan Herrmann

January 28, 2014 at 18:16

Posted in Eclipse, Object Teams, OTEquinox

Tagged with , , ,

Compiling OT/Equinox projects using Tycho

with 3 comments

In a previous post I showed how the tycho-compiler-jdt Maven plug-in can be used for compiling OT/J code with Maven.

Recently, I was asked how the same can be done for OT/Equinox projects. Given that we were already using parts from Tycho, this shouldn’t be so difficult, right?

Once you know the solution, finding the solution is indeed easy, also in this case. Here it is:

We use almost the same declaration as for plain OT/J applications:

    <pluginManagement>
	<plugin>
	    <groupId>org.eclipse.tycho</groupId>
	    <artifactId>tycho-compiler-plugin</artifactId>
	    <version>${tycho.version}</version>
	    <extensions>true</extensions>
	    <dependencies>
		<dependency>
		    <groupId>org.eclipse.tycho</groupId>
		    <artifactId>tycho-compiler-jdt</artifactId>
		    <version>${tycho.version}</version>
		    <exclusions>
			<!-- Exclude the original JDT/Core to be replaced by the OT/J variant: -->
			<exclusion>
			    <groupId>org.eclipse.tycho</groupId>
			    <artifactId>org.eclipse.jdt.core</artifactId>
			</exclusion>
		    </exclusions>
		</dependency>
		<dependency>
		    <!-- plug the OT/J compiler into the tycho-compiler-jdt plug-in: -->
		    <groupId>org.eclipse</groupId>
		    <artifactId>objectteams-otj-compiler</artifactId>
		    <version>${otj.version}</version>
		</dependency>
	    </dependencies>
	</plugin>
    </pluginManagement>

So, what’s the difference? In both cases we need to adapt the tycho-compiler-jdt plug-in because that’s where we replace the normal JDT compiler with the OT/J variant. However, for plain OT/J applications tycho-compiler-jdt is pulled in as a dependency of maven-compiler-plugin and must be adapted on this path of dependencies, whereas in Tycho projects tycho-compiler-jdt is pulled in from tycho-compiler-plugin. Apparently, the exclusion mechanism is sensitive to how exactly a plug-in is pulled into the build. Interesting.

Once I figured this out, I created and published a new version of our Maven support for Object Teams: objectteams-parent-pom:2.1.1 — publishing Maven support for Object Teams 2.1.1 was overdue anyway 🙂

With the updated parent pom, a full OT/Equinox hello world pom now looks like this:

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
 
    <parent>
        <!-- We use Object Teams: -->
        <groupId>org.eclipse</groupId>
        <artifactId>objectteams-parent-pom</artifactId>
        <version>2.1.1</version>
    </parent>
 
    <artifactId>OTEquinox-over-tycho</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <!-- We are building an Eclipse plug-in (or OSGi bundle): -->
    <packaging>eclipse-plugin</packaging>
 
    <repositories>
 
        <!-- This is where we get all Object Teams stuff from: -->
        <repository>
            <id>ObjectTeamsRepository</id>
            <name>Object Teams Repository</name>
            <url>http://download.eclipse.org/objectteams/maven/3/repository</url>
        </repository>
 
        <!-- Add any p2 repositories needed for your application, e.g.: -->
        <repository>
            <id>Juno</id>
            <name>Eclipse Juno Repository</name>
            <url>http://download.eclipse.org/releases/juno</url>
            <layout>p2</layout>
        </repository>
    </repositories>
 
    <build>
        <plugins>
            <plugin>
                <!-- We build this project using Tycho: -->
                <groupId>org.eclipse.tycho</groupId>
                <artifactId>tycho-maven-plugin</artifactId>
                <extensions>true</extensions>
            </plugin>
        </plugins>
    </build>
</project>

Looks pretty straight forward, right?

To see the full OT/Equinox Hello World Example configured for Maven/Tycho simply import OTEquiTycho.zip as a project into your workspace.

cheers,
Stephan

Written by Stephan Herrmann

October 31, 2012 at 00:16

Between the Times

with 4 comments

This week is (supposed to be) “quiet week” at Eclipse: Release Candidate 4 is in place and more testing is being done before the final Indigo Release. Time to speak of s.t. that the Object Teams project has neglected a bit in the past: compiling / building with Maven.

Edit (June 2014): In a good tradition of using quiet week I’ve updated the Maven support for OT/J 2.3.0 (Luna).

Compiling OT/J with Maven

In the days of Maven2, pluging-in a non-standard compiler required fiddling with the nexus compiler plugin, which was buggy and all that. With the recent development around Maven3 and Tycho things became easier. The following pom-snippet is all you need to tell Maven to use the OT/J compiler:

<build>
    <pluginManagement>
        <plugins>
            <plugin>
                <!-- Use compiler plugin with tycho as the adapter to the OT/J compiler. -->
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                    <compilerId>jdt</compilerId>
                </configuration>
                <dependencies>
                    <!-- This dependency provides the implementation of compiler "jdt": -->
                    <dependency>
                        <groupId>org.sonatype.tycho</groupId>
                        <artifactId>tycho-compiler-jdt</artifactId>
                        <version>${tycho.version}</version>
                        <exclusions>
                            <!-- Exclude the original JDT/Core to be replaced by the OT/J variant: -->
                            <exclusion>
                                <groupId>org.sonatype.tycho</groupId>
                                <artifactId>org.eclipse.jdt.core</artifactId>
                            </exclusion>
                        </exclusions>
                    </dependency>
                    <dependency>
                        <!-- plug the OT/J compiler into the tycho-compiler-jdt plug-in: -->
                        <groupId>org.eclipse</groupId>
                        <artifactId>objectteams-otj-compiler</artifactId>
                        <version>${otj.version}</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </pluginManagement>
</build>
 
Edit (June 2014): As of version 2.3.0 (Luna) the groupId has been changed to org.eclipse.objectteams (meanwhile, 3-part group IDs are the de-facto standard for Eclipse projects).

This relies on the fact, that the OT/J compiler is compatible with the JDT compiler, good.

To make things go smooth in various build phases a few more bits are required:

  • provide the OT/J runtime jar and its dependency (bcel)
  • tell surefire about required command line arguments for running OT/J programs

These basics are collected in a parent pom so that all you need are these two snippets:

<repositories>
    <repository>
        <id>ObjectTeamsRepository</id>
        <id>ObjectTeamsRepository</id>
        <name>Object Teams Repository</name>
        <url>http://download.eclipse.org/objectteams/maven/3/repository</url>
    </repository>
</repositories>
<parent>
    <groupId>org.eclipse</groupId>
    <artifactId>objectteams-parent-pom</artifactId>
    <version>2.0.0</version>
</parent>
 
Edit (June 2014): Again: as of version 2.3.0 the groupId is org.eclipse.objectteams.

This converts any Java project into an OT/J project, simply enough, huh?

At least compiling and testing works out-of-the box. Let me know what additional sophistication you need and if adjustments to other build phases are needed.

OT/J over m2e

Well, I’m not much of a Maven expert, but I’m an Eclipse enthusiast, so, how can we use the above inside the IDE?

The bad news: the m2e plug-in has hardcoded its support for the compiler plugin to use “javac” as the compiler. Naive attempts to use tycho-compiler-jdt have failed with the infamous "Plugin execution not covered by lifecycle configuration".

The good news: By writing a tiny OT/Equinox plug-in I was able to remedy those issues. If you have the m2e plug-in installed (Indigo version), the normal “Update Project Configuration” action will suffice:
Update Project Configuration

After that, the project is recognized as an OT/J project, both by the IDE …

… and also by maven …
Running mvn test

Where to get it?

The mentioned plug-in is brand-new and as such it is not part of the regular OTDT distribution. OTOH, the plug-in is so simple, I don’t expect a lot of changes needed. So, if you have an Indigo-ish Eclipse just add this update site:

Then install “Maven Integration for Object Teams (Early Access)”:
Install the Maven Integration for Object Teams

Don’t forget the <repository> and <parent> definitions from above and you should be ready to go! Yesss, sometimes it pays off that OT/J is a kind of Java, a lot of the tooling can just piggy-back on what’s already there for Java 🙂

Written by Stephan Herrmann

June 14, 2011 at 14:56

Posted in Eclipse, Object Teams

Tagged with , , ,