The Object Teams Blog

Adding team spirit to your objects.

Archive for the ‘OTEquinox’ Category

Runtime Specialization – At Last

with one comment

Between a rock and a hard place

Not long ago, I had to pull Object Teams out of the Eclipse simultaneous release train. Reason: the long standing issue of using BCEL for bytecode weaving, for which no Java 8 compatible version has yet been released. With the Eclipse platform moving to Java 8, this had escalated to a true blocker. During the last weeks I investigated two options in parallel:

  • Upgrade BCEL to a release candidate of the upcoming 6.0 release.
  • Finalize the alternative weaver for OT/J, which is based on ASM, and thus already capable of handling Java 8 byte codes

I soon found out that even BCEL 6.0 will not be a full solution, because it still has no real support for creating StackMapTable attributes for newly generated bytecode, which however is strictly mandatory for running on a JVM 8.

For that reason I then focussed on the OTDRE, the Object Teams Dynamic Runtime Environment. This has been announced long ago, and after all, I promised to show a sneak preview of this feature in my presentation at EclipseCon Europe:

Runtime Specialization
Java has never been so dynamic before

Success at last

Today I can report success in two regards:

  • The Object Teams Development Tooling, which itself is a complex OT/J application, can (mostly) run on the new runtime!
  • I created a first demo example that shows the new capability of runtime weaving in action – it works! 🙂

This is a major milestone! Running OTDT on top of OTDRE is a real stress test for that new component – once again I realize that dog-fooding an entire IDE on its own technology is quite an exciting exercise. While a few exceptions need to be ironed out before the Neon release, I’m confident now, that we’re finally and really on the home stretch of this effort.

But OTDRE is not just a replacement for the traditional runtime, it is also way cooler, as the second success story shows: it is indeed possible now, to throw new teams and roles into a running application. If the application has been generically prepared for this task, the new teams can be automatically activated and immediately start adapting the running application – no specific preplanning needed. With this we are able to achieve a level of dynamism that typically is only possible with dynamic languages. And all this without any compromise to static typing and analysability. This is probably too cool to be allowed.

And after all the hard work on Java 8, also OT/J can finally fully leverage the new version, not only in theory, but also in bytecode.

Less than one week to finalize the presentation. You can be sure this will be a fresh story. Join me on Wednesday, Nov 4,  in Ludwigsburg:

Runtime Specialization - Java has never been so dynamic before -- Session at EclipseCon Europe 2015

PS: The “traditional” Object Teams Runtime Environment isn’t dead, yet. I really want to keep it as an option, because both variants (OTRE / OTDRE) have quite different characteristics, and after all this component has matured over more than 10 years. But with one option already (mostly) working, I can probably wait until a proper release of BCEL 6.0, and still have it back in game before the Neon release.

Written by Stephan Herrmann

October 27, 2015 at 23:05

How to kick the fly-shuttle

leave a comment »

This post was originally written on August 6th, 2013, but got lost when moving this blog. Luckily I could recover it from backup.

Software composition by “weaving”

Programming languages in the wider field of Aspect Oriented Software Development introduce some kinds of composition concepts, that cannot directly be mapped to pure Java. We got used to speaking of “weaving” as a mechanism for preparing classes in these languages for execution on the JVM, whereas a true aspect oriented virtual machine would natively perform this composition transparently and more efficiently. But with the dominance of the JVM it will be “weaving” for a long time to come.

fly-shuttle

Back in the early days, AOSD languages performed some static composition on source or class files. When OT/J entered the stage, first frameworks emerged that allowed us to hook into the class loading process, so we could do the weaving dynamically on-the-fly, without ever storing the woven classes on disk. Work is (still) in the pipe-line to support even runtime-(re)weaving.

One theme in all the development of Object Teams is: first solve the dynamic case, that’s where the challenges lie, and leave more static cases for later as an optimization. “Later” has come now.

How (not) to hook into class loading?

A little while into the development of OT/J, Java 5 was published introducing the JPLIS API and the concept of a -javaagent. Today many instrumentation tools use this facility and OT/J was among the first on this boat. So for standalone Java applications the problem is solved.

Later, while developing Eclipse plug-ins, we noticed that it wouldn’t be overly cool if you have to invoke Eclipse with a -javaagent. Not only is this very inconvenient for installation, it is also problematic (e.g., for performance) to feed all classes of Eclipse through our weaver. We were lucky again, and the Equinox Adaptor Hooks emerged right when we needed them. With these hooks we were able to integrate OT/J in a way that enables all the on-the-fly flexibility we want while at the same time it lets us control the process so that the flexibility cannot be abused. So in Equinox (or better: OT/Equinox) the problem is (more than) solved.

The fact that Eclipse Luna will no longer support these hooks was slightly alarming news for Object Teams, but that problem is already essentially solved – details to be reported soon.

I’ve heard about successful experiments running OT/J code in tomcat, but, hey, the list of application servers/containers is long, and if we need specifically crafted integration for each environment, it’s kind of difficult to argue that OT/J runs “everywhere”. Is there a generic why to kick the fly-shuttle?

fly-shuttle-2

Dynamic weaving is cool, but not having a fallback is uncool, so I finally developed the static scenario as a special case of the dynamic one: a build-time weaver for OT/J.

The little new tool is ridiculously simple: we have a weaver capable of load-time weaving. All I had to do is: create a little utility that would invoke this weaver ahead-of-time, i.e., during building.

While I’m not a big fan of Maven, in my day job Maven is a must, so I bit the bullet used the opportunity to learn a bit about development of Maven plug-ins. I was amazed to find lots of tutorials still suggesting to use API that are deprecated in Maven 3 and more surprised to read forum threads suggesting to still use that deprecated API, partly to be compatible with Maven 2 (OK?) and partly because deprecation doesn’t always mean there is an alternative (OOPS?). It’s also interesting to see, that much fundamental, up-to-date information is not available from a central source but only from tribal knowledge in hundreds of forums, blogs and whatnots. — Enough whining, on to the new stuff.

For simple projects, all you need now is a pom that looks about like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 
    <modelVersion>4.0.0</modelVersion>
 
    <parent>
        <groupId>org.eclipse.objectteams</groupId>
        <artifactId>objectteams-parent-pom</artifactId>
        <version>2.2.0</version>
    </parent>
 
    <artifactId>OTStopwatch_Built-Time_Weaver_Example</artifactId>
    <version>0.0.1-SNAPSHOT</version>
 
    <properties>
        <tycho.version>0.18.0</tycho.version>
        <otj.version>2.2.0</otj.version>
    </properties>
 
    <build>
        <plugins>
            <plugin>
                <groupId>org.eclipse.objectteams</groupId>
                <artifactId>objectteams-weaver-maven-plugin</artifactId>
                <version>0.8.0-SNAPSHOT</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>weave</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <teamClasses>
                        <teamClass>org.eclipse.objectteams.example.stopwatch.WatchUI</teamClass>
                        <teamClass>org.eclipse.objectteams.example.stopwatch.WatchUIAnalog</teamClass>
                    </teamClasses>
                </configuration>
            </plugin>
        </plugins>
    </build>
 
    <repositories>
        <repository>
            <id>ObjectTeamsRepository</id>
            <name>Object Teams Repository</name>
            <url>http://download.eclipse.org/objectteams/maven/3/repository</url>
        </repository>
    </repositories>
 
   <dependencies>
       <dependency>
           <groupId>org.eclipse.objectteams</groupId>
           <artifactId>objectteams-runtime</artifactId>
           <version>${otj.version}</version>
       </dependency>
   </dependencies>
</project>
 

Here we enable OT/J compilation via the parent pom, specify a repo where to get the OT artifacts, a dependency and a few versions, nothing sophisticated.

What’s new is the objectteams-weaver-maven-plugin, which is minimally configured just by specifying a list of team classes that should be woven. Oops, that’s already it.

By integrating this into your build, you’ll produce class files that can be directly executed “everywhere”, i.e., without the OT/J load-time weaver, and thus: without fiddling with the class loading process. One caveat remains: naturally, by producing the woven class files offline, it will be your responsibility to correctly feed the correct versions of class files into the classpath in the desired order. If you can’t use the load-time weaver, it won’t be able to help keep things simple.

More configuration options are documented on the plug-in’s site and more explanation using our favorite examples can be found in the wiki.

Here you have it: static weaving falls of naturally if you already have dynamic weaving. And the resulting class files can truly be run “everywhere”.

I’d be curious to hear of the first OT/J apps on Android (though personally I’d prefer ubuntu).

Written by Stephan Herrmann

January 28, 2014 at 18:16

Posted in Eclipse, Object Teams, OTEquinox

Tagged with , , ,

Object Teams in the times of Eclipse Luna and Java 8

with one comment

With Eclipse Luna approaching milestone 5 it’s time to write a bit about what this year will bring for Object Teams.

Overhaul of the OT/Equinox weaver

The Luna cycle started with a surprise (for me): Equinox no longer provides the Adaptor Hooks, which allowed OT/Equinox to hook into the OSGi class loading process and perform its bytecode weaving. Outch!

On the other hand, OSGi meanwhile has a standard WeavingHook for doing this kind of stuff. With some efforts (and lots of help from Tom W. – no help from p2, though) I managed to migrate OT/Equinox to this new standard. At the end of the day, this will bring better encapsulation and hopefully better launch performance, too. Details to be posted some time later.

Lift OT/J to be based on Java 8

Some may have noticed that most my Eclipse-time is currently being spent in helping JDT to get ready for Java 8. And sure this is an exciting endeavour to be part of!

For Object Teams the cool part about this is: I’ve seen the changes for Java 8 happen in JDT, which greatly helps to adopt these changes for OT/J. Once in a while this even reveals a bug in JDT before it ever surfaced 🙂

The integration of OT/J with Java 8 still has some regressions, but my plan is to have this at good “milestone quality” when Java 8 is released in March and to get it to full release quality for Luna GA in June.

Question: does it actually make sense to combine lambdas and type annotations with roles and teams? I strongly believe it does, because these are language improvements in entirely different categories:

  • Lambda expressions help to implement algorithms in a more concise and also more modular way – this lets you think differently about functions.
  • Type annotations help enhance safety when used together with better program analysis (like, e.g., enhanced null analysis) – these let you think differently about types.
  • Roles and teams help improve the program structure at a larger scale – these let you think differently about classes and objects.

So, if each of these additions makes sense, then combining them all in one language will certainly yield a very powerful language. Some examples of combination to follow as we approach the release.

ObjectTeams.org is down, long live ObjectTeams.org

Not all material from the 10+ years of history of Object Teams has moved to Eclipse.org. Notably the language definition (OTJLD) and scientific publications are still hosted on objectteams.org. Until recently, my former University kindly provided the host for publishing these research results. Now that server has gone off service and for a while objectteams.org was dead — but as of today the relevant content is back online – sorry for the disruption. And, btw, this is also the reason why this blog has changed its URL.

Please wish me luck for the work ahead, both on JDT and Object Teams 🙂

Written by Stephan Herrmann

January 21, 2014 at 22:37

Compiling OT/Equinox projects using Tycho

with 3 comments

In a previous post I showed how the tycho-compiler-jdt Maven plug-in can be used for compiling OT/J code with Maven.

Recently, I was asked how the same can be done for OT/Equinox projects. Given that we were already using parts from Tycho, this shouldn’t be so difficult, right?

Once you know the solution, finding the solution is indeed easy, also in this case. Here it is:

We use almost the same declaration as for plain OT/J applications:

    <pluginManagement>
	<plugin>
	    <groupId>org.eclipse.tycho</groupId>
	    <artifactId>tycho-compiler-plugin</artifactId>
	    <version>${tycho.version}</version>
	    <extensions>true</extensions>
	    <dependencies>
		<dependency>
		    <groupId>org.eclipse.tycho</groupId>
		    <artifactId>tycho-compiler-jdt</artifactId>
		    <version>${tycho.version}</version>
		    <exclusions>
			<!-- Exclude the original JDT/Core to be replaced by the OT/J variant: -->
			<exclusion>
			    <groupId>org.eclipse.tycho</groupId>
			    <artifactId>org.eclipse.jdt.core</artifactId>
			</exclusion>
		    </exclusions>
		</dependency>
		<dependency>
		    <!-- plug the OT/J compiler into the tycho-compiler-jdt plug-in: -->
		    <groupId>org.eclipse</groupId>
		    <artifactId>objectteams-otj-compiler</artifactId>
		    <version>${otj.version}</version>
		</dependency>
	    </dependencies>
	</plugin>
    </pluginManagement>

So, what’s the difference? In both cases we need to adapt the tycho-compiler-jdt plug-in because that’s where we replace the normal JDT compiler with the OT/J variant. However, for plain OT/J applications tycho-compiler-jdt is pulled in as a dependency of maven-compiler-plugin and must be adapted on this path of dependencies, whereas in Tycho projects tycho-compiler-jdt is pulled in from tycho-compiler-plugin. Apparently, the exclusion mechanism is sensitive to how exactly a plug-in is pulled into the build. Interesting.

Once I figured this out, I created and published a new version of our Maven support for Object Teams: objectteams-parent-pom:2.1.1 — publishing Maven support for Object Teams 2.1.1 was overdue anyway 🙂

With the updated parent pom, a full OT/Equinox hello world pom now looks like this:

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
 
    <parent>
        <!-- We use Object Teams: -->
        <groupId>org.eclipse</groupId>
        <artifactId>objectteams-parent-pom</artifactId>
        <version>2.1.1</version>
    </parent>
 
    <artifactId>OTEquinox-over-tycho</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <!-- We are building an Eclipse plug-in (or OSGi bundle): -->
    <packaging>eclipse-plugin</packaging>
 
    <repositories>
 
        <!-- This is where we get all Object Teams stuff from: -->
        <repository>
            <id>ObjectTeamsRepository</id>
            <name>Object Teams Repository</name>
            <url>http://download.eclipse.org/objectteams/maven/3/repository</url>
        </repository>
 
        <!-- Add any p2 repositories needed for your application, e.g.: -->
        <repository>
            <id>Juno</id>
            <name>Eclipse Juno Repository</name>
            <url>http://download.eclipse.org/releases/juno</url>
            <layout>p2</layout>
        </repository>
    </repositories>
 
    <build>
        <plugins>
            <plugin>
                <!-- We build this project using Tycho: -->
                <groupId>org.eclipse.tycho</groupId>
                <artifactId>tycho-maven-plugin</artifactId>
                <extensions>true</extensions>
            </plugin>
        </plugins>
    </build>
</project>

Looks pretty straight forward, right?

To see the full OT/Equinox Hello World Example configured for Maven/Tycho simply import OTEquiTycho.zip as a project into your workspace.

cheers,
Stephan

Written by Stephan Herrmann

October 31, 2012 at 00:16

Object Teams 2.1 Milestone 7 (finally) brings hot code replacement

with 2 comments

As part of the Juno Milestone 7 also Object Teams has delivered its Milestone 7.

As the main new feature in this milestone hot code replacement finally works when debugging OT/J or OT/Equinox applications.

What took us so long to make it work?

  • Well, hot code replacement didn’t work out of the box because our load-time weaver only worked the first time each class was loaded. When trying to redefine a class in the VM the weaver was not called and thus class signatures could differ between the first loaded class and the redefined version. The VM would then reject the redefinition due to that signature change.
  • Secondly, we didn’t address this issue earlier, because I suspected this would be pretty tricky to implement. When I finally started to work on it, reality proved me wrong: the fix was actually pretty simple 🙂

In fact the part that makes it work even in an Equinox setting is so generic that I proposed to migrate the implementation into either Equinox or PDE/Debug, let’s see if there is interest.

Now when you debug any Object Teams application, your code changes can be updated live in the running debug target – no matter if you are changing teams, roles or base classes. Together with our already great debugging support, this makes debugging Object Teams programs still faster.

More new features can be found in the New&Noteworthy (accumulated since the Indigo release).

Written by Stephan Herrmann

May 12, 2012 at 22:10

Posted in Eclipse, Object Teams, OTDT, OTEquinox

Tagged with , ,

Using git for an interactive video – Replay the JDT tutorial

with 2 comments

When preparing the tutorial How to train the JDT dragon, Ayushman and me packed way more material than we could possible teach in 3 hours time.

Luckily, Ayush came up with an idea that enables all of you to go through the exercise all on your own: instead of creating zip files with various stages of the example projects, we pushed everything to github. I apologize to those who were overwhelmed by how we pushed git onto the tutorial participants in Reston.

Now, as EclipseCon is over, here’s what you can do:

  • Fetch the slides: PDF or slideshare
  • Clone the material for the plain JDT part (git: URL), containing two projects for the first two exercises.
    • org.eclipsecon2012.jdt.tutorial1: traverse the Java Model starting from the element selected in the UI
    • org.eclipsecon2012.jdt.quickFix1: implement a quickfix for adding a missing null check
  • Clone the misc repo (git: URL), containing
    • handouts: instructions for the exercises
    • org.eclipsecon2012.jdt.populator: plug-in that was used in the tutorial to create sample content in the runtime workbench; show cases how to programmatically create Java files from String content
  • Clone the OT repo (git: URL), containing
    • AntiDemo: the OT/Equinox hello-world: prohibition of class names starting with “Foo”.
    • Reachability: a full blown reachability analysis in < 300 LOC.

The last exercise, “Reachability”, we couldn’t even show in Reston, but luckily this is the most detailed repo: in 13+1 commits I recorded the individual steps of adding inter-procedural reachability analysis piggy-back on top of the JDT compiler. If you replay these commits from the git repo you can watch me creating this little plug-in from scratch. Starting from step 5 you’ll be able to see the results when you fire up a runtime workbench: the analysis will print to stdout what methods it found during each full build.

Look at the commit comments which summarize what I would have explained in demo mode. Still the material is pretty dense as it explains three technologies at the same time:

  1. How does this analysis work?
  2. What points in the JDT do we hook onto?
  3. How is the integration achieved using Object Teams.

Speaking of integration: using the “Binding Editor” you can see all bindings in a tabular view:

Bindings of the Reachability plug-in

The right hand side of the table give the full list of JDT elements we connect to:

  • classes that are extended using a role_obj role
  • methods/fields that are accessed externally via a callout binding callout binding
  • methods that are intercepted using a callin binding (after) callin binding.

As a special bonus, step 14 in the git repo shows how to add problem markers for each unreachable method, so that the JDT will actually offer its existing quickfix for removing:
Unreachable methods
(Yep, both methods are detected as unreachable: calling each other without being called from another reachable method doesn’t help).

I hope you enjoy playing with the examples. For questions please use one of the forums:

Written by Stephan Herrmann

April 7, 2012 at 15:20

Mix-n-match language support

leave a comment »

I’ve been involved in the release of different versions of the JDT lately, supporting different flavors of Java.

Classical release management

At the core we have the plain JDT, of which we published the 3.7.0 release in June and right now first release candidates are being prepared towards the 3.7.1 service release, which will be the first official release to support Java 7. At the same time the first milestones towards 3.8 are being built. OK, this is almost normal business — with the exception of the service release differs more than usual from its base release, due to the unhappy timing of the release of Java 7 vs. Eclipse 3.7.

So that’s 3 versions in 2 month’s time.

First variant: Object Teams

The same release plan is mirrored by the Object Teams releases 2.0.0, 2.0.1RC1, 2.1.0M1. Merging the delta from JDT 3.7 to 3.7.1 into the OTDT was a challenge, given that this delta contained the full implementation of all that’s new in Java 7. Still with the experience of regularly merging JDT/Core changes into the OT variant, the pure merging was less than one day plus a couple more days until all 50000+ tests were green again. The nice thing about the architecture of the OTDT: after merging the JDT/Core, I was done. Since all other adaptations of the JDT are implemented using OT/Equinox adopting, e.g., all the new quick assists for Java 7 required a total of zero minutes integration time.

I took the liberty of branching 2.0.x and 2.1 only after integrating the Java 7 support, which also means that 2.1 M1 has only a small number of OT-specific improvements that did not already go into 2.0.1.

This gives 6 versions of the JDT in 2 month’s time.

Prototyping annotation based null analysis

As I wrote before, I’m preparing a comprehensive new feature for the JDT/Core: static analysis for potential NullPointerException based on annotations in the code. The latest patch attached to the bug had almost 3000 lines. Recent discussions at ECOOP made me change my mind in a few questions, so I changed some implementation strategies. Luckily the code is well modularized due to the use of OT/Equinox.

Now came the big question: against which version of the JDT should I build the null-annotation add-on? I mean, which of the 6 versions I have been involved in during the last 2 months?

As I like a fair challenge every now and then I decided: all six, i.e., I wanted to support adding the new static analysis to all six JDT versions mentioned before.

Integration details

Anybody who has worked on a Java compiler will confirm: if you change one feature of the compiler chances are that any other feature can be broken by the change (I’m just paraphrasing: “it’s complex”). And indeed, applying the nullity plug-in to the OTDT caused some headache at first, because both variants of the compiler make specific assumptions about the order in which specific information is available during the compilation process. It turned out that two of these assumptions where simply incompatible, so I had to make some changes (here I made the null analysis more robust).

At the point where I thought I was done, I tripped over an ugly problem that’s intrinsic to Java.
The nullity plug-in adapts a method in the JDT/Core which contains the following switch statement:

        while ((token = this.scanner.getNextToken()) != TerminalTokens.TokenNameEOF) {
                IExtendedModifier modifier = null;
                switch(token) {
                        case TerminalTokens.TokenNameabstract:
                                modifier = createModifier(Modifier.ModifierKeyword.ABSTRACT_KEYWORD);
                                break;
                        case TerminalTokens.TokenNamepublic:
                                modifier = createModifier(Modifier.ModifierKeyword.PUBLIC_KEYWORD);
                                break;
                        // more cases
                }
        }
 

I have a copy of this method where I only added a few lines to one of the case blocks.
Compiles fine against any version of the JDT. But Eclipse hangs when I install this plugin on top of a wrong JDT version. What’s wrong?

The problem lies in the (internal) interface TerminalTokens. The required constants TokenNameabstract etc. are of course present in all versions of this interface, however the values of these constants change every time the parser is generated anew. If constants were really abstractions that encapsulate their implementation values, all would be fine, but the Java byte code knows nothing about such an abstraction, all constant values are inlined during compilation. In other words: the meaning of a constant depends solely on the definitions which the compiler sees during compilation. Thus compiling the above switch statement hardcodes a dependency on one particular version of the interface TerminalTokens. BAD.

After recognizing the problem, I had to copy some different versions of the interface into my plug-in, implement some logic to translate between the different encodings and that problem was solved.

What’s next?

Nothing is next. At this point I could apply the nullity plug-in to all six versions of the JDT and all are behaving well.

We indeed have 12 versions of the JDT in 2 month’s time.

Mix-n-match

Would you like Java with our without the version 7 enhancements (stable release or milestone)? May I add some role and team classes? How about a dash more static analysis? It turns out we have more than just one product, we have a full little product line with features to pick or opt-out:

 

Java 6

Java 7
Indigo

Indigo SR1

Juno M1
no null annotations

Plain JDT

 

 

 
OTDT

 

 

 
with null annotations

Plain JDT

 

 

 
OTDT

 

 

 

Just make your choice 🙂
Happy hacking with null annotations and try-with-resources in OT/J.

EclipseCon Europe 2011  

BTW: if you want to hear a bit more about the work on null annotations, you should really come to EclipseCon Europe — why not drop a comment at this submission 🙂

Written by Stephan Herrmann

August 19, 2011 at 17:22