The Object Teams Blog

Adding team spirit to your objects.

Posts Tagged ‘install

How to kick the fly-shuttle

leave a comment »

This post was originally written on August 6th, 2013, but got lost when moving this blog. Luckily I could recover it from backup.

Software composition by “weaving”

Programming languages in the wider field of Aspect Oriented Software Development introduce some kinds of composition concepts, that cannot directly be mapped to pure Java. We got used to speaking of “weaving” as a mechanism for preparing classes in these languages for execution on the JVM, whereas a true aspect oriented virtual machine would natively perform this composition transparently and more efficiently. But with the dominance of the JVM it will be “weaving” for a long time to come.

fly-shuttle

Back in the early days, AOSD languages performed some static composition on source or class files. When OT/J entered the stage, first frameworks emerged that allowed us to hook into the class loading process, so we could do the weaving dynamically on-the-fly, without ever storing the woven classes on disk. Work is (still) in the pipe-line to support even runtime-(re)weaving.

One theme in all the development of Object Teams is: first solve the dynamic case, that’s where the challenges lie, and leave more static cases for later as an optimization. “Later” has come now.

How (not) to hook into class loading?

A little while into the development of OT/J, Java 5 was published introducing the JPLIS API and the concept of a -javaagent. Today many instrumentation tools use this facility and OT/J was among the first on this boat. So for standalone Java applications the problem is solved.

Later, while developing Eclipse plug-ins, we noticed that it wouldn’t be overly cool if you have to invoke Eclipse with a -javaagent. Not only is this very inconvenient for installation, it is also problematic (e.g., for performance) to feed all classes of Eclipse through our weaver. We were lucky again, and the Equinox Adaptor Hooks emerged right when we needed them. With these hooks we were able to integrate OT/J in a way that enables all the on-the-fly flexibility we want while at the same time it lets us control the process so that the flexibility cannot be abused. So in Equinox (or better: OT/Equinox) the problem is (more than) solved.

The fact that Eclipse Luna will no longer support these hooks was slightly alarming news for Object Teams, but that problem is already essentially solved – details to be reported soon.

I’ve heard about successful experiments running OT/J code in tomcat, but, hey, the list of application servers/containers is long, and if we need specifically crafted integration for each environment, it’s kind of difficult to argue that OT/J runs “everywhere”. Is there a generic why to kick the fly-shuttle?

fly-shuttle-2

Dynamic weaving is cool, but not having a fallback is uncool, so I finally developed the static scenario as a special case of the dynamic one: a build-time weaver for OT/J.

The little new tool is ridiculously simple: we have a weaver capable of load-time weaving. All I had to do is: create a little utility that would invoke this weaver ahead-of-time, i.e., during building.

While I’m not a big fan of Maven, in my day job Maven is a must, so I bit the bullet used the opportunity to learn a bit about development of Maven plug-ins. I was amazed to find lots of tutorials still suggesting to use API that are deprecated in Maven 3 and more surprised to read forum threads suggesting to still use that deprecated API, partly to be compatible with Maven 2 (OK?) and partly because deprecation doesn’t always mean there is an alternative (OOPS?). It’s also interesting to see, that much fundamental, up-to-date information is not available from a central source but only from tribal knowledge in hundreds of forums, blogs and whatnots. — Enough whining, on to the new stuff.

For simple projects, all you need now is a pom that looks about like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 
    <modelVersion>4.0.0</modelVersion>
 
    <parent>
        <groupId>org.eclipse.objectteams</groupId>
        <artifactId>objectteams-parent-pom</artifactId>
        <version>2.2.0</version>
    </parent>
 
    <artifactId>OTStopwatch_Built-Time_Weaver_Example</artifactId>
    <version>0.0.1-SNAPSHOT</version>
 
    <properties>
        <tycho.version>0.18.0</tycho.version>
        <otj.version>2.2.0</otj.version>
    </properties>
 
    <build>
        <plugins>
            <plugin>
                <groupId>org.eclipse.objectteams</groupId>
                <artifactId>objectteams-weaver-maven-plugin</artifactId>
                <version>0.8.0-SNAPSHOT</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>weave</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <teamClasses>
                        <teamClass>org.eclipse.objectteams.example.stopwatch.WatchUI</teamClass>
                        <teamClass>org.eclipse.objectteams.example.stopwatch.WatchUIAnalog</teamClass>
                    </teamClasses>
                </configuration>
            </plugin>
        </plugins>
    </build>
 
    <repositories>
        <repository>
            <id>ObjectTeamsRepository</id>
            <name>Object Teams Repository</name>
            <url>http://download.eclipse.org/objectteams/maven/3/repository</url>
        </repository>
    </repositories>
 
   <dependencies>
       <dependency>
           <groupId>org.eclipse.objectteams</groupId>
           <artifactId>objectteams-runtime</artifactId>
           <version>${otj.version}</version>
       </dependency>
   </dependencies>
</project>
 

Here we enable OT/J compilation via the parent pom, specify a repo where to get the OT artifacts, a dependency and a few versions, nothing sophisticated.

What’s new is the objectteams-weaver-maven-plugin, which is minimally configured just by specifying a list of team classes that should be woven. Oops, that’s already it.

By integrating this into your build, you’ll produce class files that can be directly executed “everywhere”, i.e., without the OT/J load-time weaver, and thus: without fiddling with the class loading process. One caveat remains: naturally, by producing the woven class files offline, it will be your responsibility to correctly feed the correct versions of class files into the classpath in the desired order. If you can’t use the load-time weaver, it won’t be able to help keep things simple.

More configuration options are documented on the plug-in’s site and more explanation using our favorite examples can be found in the wiki.

Here you have it: static weaving falls of naturally if you already have dynamic weaving. And the resulting class files can truly be run “everywhere”.

I’d be curious to hear of the first OT/J apps on Android (though personally I’d prefer ubuntu).

Written by Stephan Herrmann

January 28, 2014 at 18:16

Posted in Eclipse, Object Teams, OTEquinox

Tagged with , , ,

Between the Times

with 4 comments

This week is (supposed to be) “quiet week” at Eclipse: Release Candidate 4 is in place and more testing is being done before the final Indigo Release. Time to speak of s.t. that the Object Teams project has neglected a bit in the past: compiling / building with Maven.

Edit (June 2014): In a good tradition of using quiet week I’ve updated the Maven support for OT/J 2.3.0 (Luna).

Compiling OT/J with Maven

In the days of Maven2, pluging-in a non-standard compiler required fiddling with the nexus compiler plugin, which was buggy and all that. With the recent development around Maven3 and Tycho things became easier. The following pom-snippet is all you need to tell Maven to use the OT/J compiler:

<build>
    <pluginManagement>
        <plugins>
            <plugin>
                <!-- Use compiler plugin with tycho as the adapter to the OT/J compiler. -->
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                    <compilerId>jdt</compilerId>
                </configuration>
                <dependencies>
                    <!-- This dependency provides the implementation of compiler "jdt": -->
                    <dependency>
                        <groupId>org.sonatype.tycho</groupId>
                        <artifactId>tycho-compiler-jdt</artifactId>
                        <version>${tycho.version}</version>
                        <exclusions>
                            <!-- Exclude the original JDT/Core to be replaced by the OT/J variant: -->
                            <exclusion>
                                <groupId>org.sonatype.tycho</groupId>
                                <artifactId>org.eclipse.jdt.core</artifactId>
                            </exclusion>
                        </exclusions>
                    </dependency>
                    <dependency>
                        <!-- plug the OT/J compiler into the tycho-compiler-jdt plug-in: -->
                        <groupId>org.eclipse</groupId>
                        <artifactId>objectteams-otj-compiler</artifactId>
                        <version>${otj.version}</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </pluginManagement>
</build>
 
Edit (June 2014): As of version 2.3.0 (Luna) the groupId has been changed to org.eclipse.objectteams (meanwhile, 3-part group IDs are the de-facto standard for Eclipse projects).

This relies on the fact, that the OT/J compiler is compatible with the JDT compiler, good.

To make things go smooth in various build phases a few more bits are required:

  • provide the OT/J runtime jar and its dependency (bcel)
  • tell surefire about required command line arguments for running OT/J programs

These basics are collected in a parent pom so that all you need are these two snippets:

<repositories>
    <repository>
        <id>ObjectTeamsRepository</id>
        <id>ObjectTeamsRepository</id>
        <name>Object Teams Repository</name>
        <url>http://download.eclipse.org/objectteams/maven/3/repository</url>
    </repository>
</repositories>
<parent>
    <groupId>org.eclipse</groupId>
    <artifactId>objectteams-parent-pom</artifactId>
    <version>2.0.0</version>
</parent>
 
Edit (June 2014): Again: as of version 2.3.0 the groupId is org.eclipse.objectteams.

This converts any Java project into an OT/J project, simply enough, huh?

At least compiling and testing works out-of-the box. Let me know what additional sophistication you need and if adjustments to other build phases are needed.

OT/J over m2e

Well, I’m not much of a Maven expert, but I’m an Eclipse enthusiast, so, how can we use the above inside the IDE?

The bad news: the m2e plug-in has hardcoded its support for the compiler plugin to use “javac” as the compiler. Naive attempts to use tycho-compiler-jdt have failed with the infamous "Plugin execution not covered by lifecycle configuration".

The good news: By writing a tiny OT/Equinox plug-in I was able to remedy those issues. If you have the m2e plug-in installed (Indigo version), the normal “Update Project Configuration” action will suffice:
Update Project Configuration

After that, the project is recognized as an OT/J project, both by the IDE …

… and also by maven …
Running mvn test

Where to get it?

The mentioned plug-in is brand-new and as such it is not part of the regular OTDT distribution. OTOH, the plug-in is so simple, I don’t expect a lot of changes needed. So, if you have an Indigo-ish Eclipse just add this update site:

Then install “Maven Integration for Object Teams (Early Access)”:
Install the Maven Integration for Object Teams

Don’t forget the <repository> and <parent> definitions from above and you should be ready to go! Yesss, sometimes it pays off that OT/J is a kind of Java, a lot of the tooling can just piggy-back on what’s already there for Java 🙂

Written by Stephan Herrmann

June 14, 2011 at 14:56

Posted in Eclipse, Object Teams

Tagged with , , ,

A Short Train Ride

with 2 comments

Less than a week ago I happily announced that Object Teams is on the Indigo Train.
Much water has gone under the bridge since then and the above statement is history.

Events where triggered by what was actually a little bug in the b3 aggregator. Only by way of this bug some people noticed that there was a ”’patch feature”’ inside the repository, i.e., a feature (“Object Teams Patch for JDT/Core”) that replaces the jdt.core plugin with a variant.

One part of me is very happy this bug occurred because finally an issue got the attention I had tried to raise at various occasions before. The lesson is:

Installing stuff with p2 may do other things than just adding new functionality, it may change existing stuff.

(see this post, e.g.).

The other part of me got very worried because during that debate some harsh statements occurred that would effectively amount to excluding Object Teams from Eclipse.org. That’s a little more attention than I had intended. As in any heated debate some of the arguments sounded to me more like ideology than anything that could possibly be discussed open-mindedly.

I had mixed feelings regarding the technical scope: the outcry only banned one specific technology: patch features. I don’t see how the goal to protect a project’s bits and bytes against influence from other projects can be achieved without also discussing: access to internal, byte-code weaving and – worst of all, I believe – reflection. To be perfectly open: Object Teams uses all these techniques except for reflection. Personally, I would even argue for banning projects that do use setAccessible(true), but that’s not a realistic option because then quite likely the whole Train would dissolve into a mist.

I am actually guilty of a technical simplification during this debate: I focused too much on the idea that a user would explicitly select features to install, not accounting for the possibility that the jdt.core plugin can well be pulled in invisibly due to dependencies among plugins. So, yes, if Object Teams would still be in the repository, and if a user installed a package without the JDT and if that user did never select to install the JDT and if that user selects another feature that implicitly requires the jdt.core plugin, then that user would unexpectly install the OT variant of the jdt.core. I agree that this is not ideal. I personally would have been happy to take this risk because I know how thoroughly the OT variant is tested for compatibility. And for the remaining minuscule risk I would have been happy to promise same-day fixes. But risk assessment naturally depends on perspective and I understand that others come to different conclusions when weighing the issues.

From two days distance I can already laugh at one implication of the central argument, paraphrased as: the JDT/Core team must be protected against harmful actions from the OT team. When spelling this out in names, one of the sentences reads: “Stephan Herrmann must be protected against harmful actions by Stephan Herrmann”. I should really be careful, because I’ll never be able to escape him!

Where to go?

  • As we’re banned from the Train, I willed hurriedly book a plane ticket to Indigo. Make sure we come by a Graduation office on the way.
  • I do hope that bug 316702 get’s sufficient attention now. Seriously: if you are so detrimentally determined about patch features, then the UI must report it. And if it reports this, it might as well report other techniques that have similar effects!
  • I appreciate any offers for helping OT/J towards a solution that avoids replacing a plugin. As of today and after more than seven years of looking at this, I see no way how this can be done, but that doesn’t mean we shouldn’t try still harder.
Unfortunately, the debate consumed all the time and energy I had planned for preparing a presentation at the EclipseCON Audition. However, Lynn finally made my day by letting me know that my submission is the lucky #42. So I’m making progress towards my all-seasons Eclipse collection 🙂

Written by Stephan Herrmann

November 18, 2010 at 14:32

Posted in Eclipse, Object Teams, OTDT

Tagged with , , ,

Moving business

leave a comment »


Remember the last time you had to cram your whole hosehold into boxes, bags and cases? You may feel excited about your new home etc. but the whole boxing business is quite a drag, ain’t it? There’s of course at least two ways of approaching this:

  1. Don’t look, just shovel everything randomly into boxes
  2. Look at each single piece, indulge in memories associates with it and sort it to its likes

Obviously, (1) means that on the other side you will unpack a whole mess of junk. OTOH, (2) won’t be finished before the moving truck arives. Still deep inside there lives a bit of hope, that you’ll move to your new home with only that stuff you’ll actually want, and everything ready to be neatly deployed into its new destination. Moving could free you from all the junk you don’t want any more, right? And even more, after the move, you may want to know, where everything is, right?

When moving the Object Teams project to Eclipse I was in the lucky situation that I could indeed use the occasion to sort through some of our stuff. The software engineer might be tempted to even speak of some “quality assurance” along the way, but let’s be careful with our wording for now.

On January 26 this year, the Eclipse Object Teams Project was created and we started to put up signs “Object Teams is moving to Eclipse.org”. Recently, I changed that sign to “Object Teams has moved to Eclipse.org”. So, what exactly happened between then and now?

Learning the infrastructure

At first the newly appointed eclipse committer and project lead is overwhelmed with all the shiny technology: web server, wiki, version repositories, build servers, download servers, the portal, project metadata, accounts for this and that, bugzilla components, versions and milestones and what-not. “Alles so schön bunt hier!”

I won’t indulge in talking about the paper work needed at this stage, after some four days I had most my accounts and the project was registered as “incubation – conforming”, so we were ready to go into the parallel IP process.

Initial contribution

On project day #12 I submitted my first CQ and, yes, that submission was already the result of heavy refactoring: I had renamed most (not all!!) occurrences of org.objectteams to org.eclipse.objectteams. A piece of cake? Well not exactly if the code piles up to a zip of 35 MBytes (not including .class files and nested archives), and not if your team-support plug-in goes berserk on some of the renamings and if … (see also this post).

Parallel IP process – our version

In fact our version of the parallel IP process looked like this:

On one thread I was chasing after some people and institutions to just provide the necessary signed statements of code provenance. Wrt the individuals this was painless, however, the university and the research institute involved both had their very specific strategies for delaying the project. All-in-all it took them more than one week per sentence in the final document. Or would you prefer the words-per-day count?   The much feared IP analysis turned out to be a very constructive collaboration with Sharon Corbett. I was really amazed about the obscure pieces of code (and comments) she brought to light, things that I never knew where in there. So that was helpful information, actually 🙂 .
Most of all I was pleased by her quick responsiveness – quite in contrast to the other thread. Thanks Sharon!
I should also thank Jan Wloka who from the outset of the project took care that we’d have copyright headers and that stuff.

The effect was: at the time I got the signatures that cleared us to check our sources into svn the IP analysis was already done and complete! And not only that, during that process we’ve done some significant cleanup.

Code cleanup triggered by the move:

  • Proviously, Object Teams used the JMangler framework for launching with our bytecode transformers in place. This was a great thing to have back in the olden Java 1.3 times. But in 2010 our Java 5 based alternative had matured and we didn’t even have to put JMangler into any of our moving boxes 🙂
  • We used to maintain a patched version of BCEL 4.4.1 (developed at Freie Universität Berlin, as the namespace de.fub still announced). I consulted the AspectJ folks who maintain a patched version, too. But their patch only vaguely resembles the original, and they clearly stated that they saw now chances of these changes being adopted upstream. So, I went back to our sources, checked the patches, checked the current version 5.2 of bcel and found that the remaining bugs could actually be easily worked around from the outside. That’s when I learned the details of Orbit: since our initial commits to Eclipse we never had to bother about the bcel version, it just comes right flying from the Orbit. So we got that legacy version cleared up.
  • My heart skipped a tiny single beat, when I learned that one of our most central data structures could not be accepted at Eclipse: I had patched class WeakHashMap from the OpenJDK to create a DoublyWeakHashMap with quite unique characteristics concerning garbage collection. We need that! Yet, the license (GPL with “classpath exception”) was not accepted. I made a quick experiment with wrapping instead of patching and guess what I learned (again): while the patched version was created in the pursuit of performance, still, after changing the strategy (to what was destined to be slower) my measurements could not show any performance penalty. So, carve that in stone: never optimize without measuring. The new version has the same performance – and no license issues!

Of course, there were more issues like needing to file a new CQ just for using files like xhtml1-strict.dtd, but those caused no grief after all. Enter the next phase.

Getting everything to build and test on eclipse servers

OK, when we opened the boxes at our new home, some of the content was actually broken on the way.

Fixing broken builds

The ugliest part was getting an ancient set of PDE/Build scripts to run on build.eclipse.org. Digging through a 30MByte build-log looking for the cause of a build failure never was fun. The point that dissatisfies me with all the build technology I’ve seen so far: You have a build that works on one machine and with one version of the software. Then, one arbitrary piece of the setup changes, let’s take a big change as the example: moving your JDK from 5 to 6. It’s OK that things may break at this point, but the kind of breakage frequently seems to have nothing to do with what you’ve changed, e.g., after moving to a different JDK the compiler can no longer resolve java.long.Object, whereas everybody knows: that’s not the difference between the two JDKs. The problem is not broken builds, the problem is how little clues the logs give you for finding the root cause of the breakage, or even: telling you how to fix it. A technology that works when it works is one thing. A technology that helps you get it to work is another (and we’re working hard to make Object Teams fall into the second category).

Modernizing the build

OK, enough complained, the move to Eclipse again gave reason to cleanup that monster build and even update to using some of the automatic built-in p2 stuff (yes, finally we use p2.gathering=true), rather than manually invoking the various p2 applications (publisher, director). When it runs, you may even get the impression you know why it does.

The final round in improving our build was adding bundle signing, yeah! Of course, that’s when all the p2-metadata generated during the build don’t help you any more because those include the checksums of your unsigned jars. So I created a tiny little shell script to automate those steps required after a successful build&test. I ended up with 7 more transformations of our metadata needed at this stage. So we’re back at directly invoking p2 publisher, p2 director, do various XSL transformations etc. Most of these could actually be done by a PDE/Build – p2 integration, but let’s not expect too much, not now.

Did I mention the almost 50000 tests that successfully run during each build? Well, that’s what we owe our users, right?

It builds – let’s ship it!

OK, let’s move ahead to the success-story-part. Less than a month after the initial commit we had our first milestone. It’s so good to be back in business 🙂 After fixing all those migration-induced regressions I’m sure our code has better quality than before.

User side migration

Only one burden we had to pass on to users: due to the changed namespace, the configuration files of existing OT/J projects have to be updated. Luckily, it wasn’t too difficult to add some specific build-path problems and quickfixes, which should make the migration for users pretty smooth.

Installing

Right while I was publishing our first milestones a new cool tool came around the corner: the Market Place Client. So, now if you download, e.g., the Helios Package “Eclipse IDE for Java Developers” you’ll get the OTDT installed without having to know the download address, just select Help > Eclipse Marketplace and search for “Object Teams”, and you’re ready to hit “Install“:

Interestingly, in order for this to work I had to ask for this feature, which later down the road triggered this blocker security issue. At the end of the day this made me ponder about generalizing various things that the user might want to know when installing software. And indeed, Object Teams should play an active role in this discussion: the whole business of OT/Equinox is based on the assumption that the user agrees with what we are doing. We already do our best in treating the users as grown-ups who can make their own decisions, if we provide sufficient
information, like:

This little screenshot tells you a whole story about this version of the bundle org.eclipse.jdt.core (see last column):

  • The icon in the 1st column says: this bundle is signed, but that signed content is going to be woven at load time before the JVM sees it
  • Columns 2 & 3 give the obvious information that this is not the version provided by the JDT team but something from the Object Teams project, which BTW. is still in its incubation phase.
  • Column 4 finally gives you all the gory details: a sophisticated version number plus the list of OT/Equinox plugins that have declared to adapt the current plugin.

That’s the kind of transparency we show upon request after the software is installed. The mentioned bug 316702 is about providing similar transparency already during install.

So, what’s the plan?

Given that all legal and technical matters have been sorted out to this point, and given that the tool is in an even better shape than the final OTDT 1.4.0 from objectteams.org, what’s our plan?

  • Just recently I requested a Release Review, tentatively scheduled for July 7, so with only little delay after Helios we should have an actual, stable release.
  • I decided to defer the project graduation some more months to give us time to define which parts of the software are actually API.

Where can I see it in action?

Well, given the current milestone releases (and the ease of installing) and all the documentation we have in the wiki nothing should stop you from running your first Object Teams demo in do-it-yourself mode 🙂

Otherwise, if you happen to be in Vienna on June 25, just come to this DemoCamp and I’ll help you to get started with Object Teams.

So, indeed for the Object Teams project that past 6 months were used to turn this:

into this:

Written by Stephan Herrmann

June 15, 2010 at 23:06

Posted in Eclipse, Object Teams, OTDT, OTEquinox

Tagged with , , ,