The Object Teams Blog

Adding team spirit to your objects.

Archive for the ‘Eclipse’ Category

Several Languages Java™ 8

with 4 comments

More than 3 years ago, on March 18, 2014, Java™ 8 was released, and on the same day Eclipse released support for this new version. I have repeatedly made the point that the Eclipse compiler for Java (ecj) as a second implementation of JLS (after javac) serves the entire Java community as a premier means for quality assurance.

Today, in April 2017, I can report that this effort is still far from complete. Still, JLS, javac and ecj do not define the exact same language. Time to take stock what these differences are about.

My own work on ecj focuses on an aspect that tries hard to remain invisible to casual users: type inference. It’s the heavy machinery behind generic methods, diamond expressions and lambdas, allowing users to omit explicit type information in many places, leaving it to the compiler to figure out the fine print.

To be honest, when we first shipped support for Java 8, I was well expecting lots of bug reports to come in, which would point out corner cases of JLS that we hadn’t implemented correctly. There was one area, that I felt particularly uneasy about: how type inference blends with overload resolution. During the Mars cycle of development Srikanth thankfully performed a major rework and clean up of this exact area.
(I can’t pass the opportunity to report what I learned from this exercise: Overloading is a huge contributor of complexity in the Java language which in (not only) my opinion doesn’t carry its own weight — not a fraction of it).

We are not done with Java 8

The fact that still 2 years after that rework we constantly receive bug reports against Java 8 type inference is unsettling in a way.

To give some numbers to it: during every 6-week milestone we fixed between 1 and 6 bugs in type inference. None of these bugs is solved in a coffee break, some compete for the title “toughest challenge I faced in my career”.

We have a total of 103 bugs explicitly marked as 1.8 inference bugs. Of these

  • 17 were resolved before Java 8 GA
  • 52 have been resolved in the three years since Java 8 GA
  • 34 are still unresolved today.

This will likely keep me busy for at least one more year.

In the early days of Java 8 we could identify two clusters where behavioral differences between javac and ecj could be observed:

  • wildcard capture
  • raw types

(I’ll have a note about the latter at the end of this post).

In these areas we could comfort ourselves by pointing to known bugs in javac. We even implemented code to conditionally mimic some of these javac bugs, but frankly, establishing bug compatibility is even more difficult than truthfully implementing a specification.

Meanwhile, in the area of wildcard capture, javac has been significantly improved. Even though some of these fixes appear only in Java 9 early access builds, not in Java 8, we can observe both compilers converging, and given that the major bugs have been fixed, it is getting easier to focus on remaining corner cases. Good.

Java “8.1”

One event almost went under our radar: In February 2015 a revised version of JLS 8 was published. As part of this update, a few sentences have been added on behalf of JDK-8038747. While the spec may be hard to grok by outsiders, the goal can be explained as enabling a compiler to include more type hints from the bodies of lambda expressions that are nested inside a generic method invocation.

In fact, collecting type information from different levels in the AST was a big goal of the type inference rewrite in Java 8, but without the 2015 spec update, compilers weren’t even allowed to look into the body of a lambda, if the lambda does not specify types for its arguments.

m(a -> a.b())
What do we know about b, while types for m and a are still unknown?

Conceptually, this is immensely tricky, because generally speaking, the code in the bodies of such type-elided lambdas can mean just about anything, while the signature of the lambda is not yet known. So we were happy about the spec update, as it promised to resolve a bunch of test cases, where javac accepts programs that ecj – without the update – was not able to resolve.

Ever since, each improved version of ecj created a regression for one or more of our dear users. We debugged no end, read between the lines of JLS, but couldn’t find a solution that would satisfy users in all cases. And they kept complaining that javac had no problems with their programs, even earlier versions of ecj accepted their program, so rejecting it now must be a regression.

“Switching the alliance”

Up-to that point, I saw our main ally in the authors of JLS, Dan Smith and Alex Buckley from Oracle. In particular Dan Smith has been a tremendous help in understanding JLS 8 and analyzing where our reading of it deviated from the authors’ intention. Together we identified not only bugs in my interpretation and implementation of JLS, but also several bugs in javac.

When we iterated bugs relating to JDK-8038747 time and again, this approach was less effective, coming to no conclusion in several cases. I slowly realized, that we were reaching a level of detail that’s actually easier to figure out when working with an implementation, than at the specification level.

This is when I started to seek advice from javac developers. Again, I received very valuable help, now mostly from Maurizio Cimadamore. Previously, my perception was, that JLS is the gold standard, and any deviation or even just liberal interpretation of it is bad. During the discussion with Maurizio I learned, that in some sense javac is actually “better” than JLS, not only in accepting more type-correct programs, but also in terms of better reflecting the intention of the JSR 335 experts.

So I started to deliberately deviate from JLS, too. Instead of “blaming” javac for deviating from JLS, I now “blame” JLS for being incomplete wrt the intended semantics.

To put this effort into proportion, please consider the figure of 103 bugs mentioned above. From these, 17 bugs have a reference to JDK-8038747. Coincidentally, this is the exact same number as those great bug reports prior to Java 8 GA, that gave us the huge boost, enabling us to indeed deliver a high quality implementation right on GA. In other words, this is a huge engineering effort, and we have no idea, how close to done we are. Will we face the next round of regressions on every new release we publish?

If you work from a specification, there is a point where you feel confident that you did all that is required. Knowing that fulfilling the spec is not enough, it’s impossible to say, what is “enough”.

What is “better”?

With wildcard captures and raw types, it was easy to argue, that certain programs must be rejected by a compiler, because they are not type safe and can blow up at runtime in unexpected locations. In the area around JDK-8038747 javac tends to accept more programs than JLS, but here it would be unreasonable to expect javac to change and start rejecting these “good” programs.

Still, calling out a competition of who accepts more “good” programs would be a bad idea, too, because this would completely abandon the goal of equivalence between compilers. After compiling with one compiler, one could never be sure that another compiler would also accept the same program. The term “Java” would loose its precise meaning.

This implies, every attempt to better align ecj with javac, based on knowledge about the implementation and not based on JLS, should be seen as a temporary workaround. To resume its role of leadership, JLS must catch up with any improvements done in the implementation(s).

To comfort the reader, I should say that in all cases discussed here, there’s always a safe fallback: when inference fails to find a solution, it is always possibly to help the compiler by adding some explicit type arguments (or argument types for a lambda). More importantly, such additions, which may be required for one compiler, should never cause problems for another compiler.

Also note, that explicit type arguments are always to be preferred over type casts (which some people tend to use as a workaround): type arguments will help for effective type checking, whereas type casts bypass type checking and can blow up at runtime.

Thanks and Sorry!

I wrote this post in the desire to reach out to our users.

First: Each reproducible bug report is highly valuable; this is what drives JDT code towards higher and higher quality. By accumulating test cases from all these reports we gradually create a test suite that provides the best available safety net.

Second: I am sorry about every regression introduced by any of our fixes, but as this post should explain, we are traveling uncharted territory: some of the corner cases we are currently addressing are not sufficiently covered by JLS. Additionally, type inference is inherently sensitive to the slightest of changes. Predicting, which programs will be affected by a given change in the implementation of type inference is near impossible.

Yet, it’s certainly not a game of “them” vs “us”: JLS, javac, and ecj, we’re all in this together, and only by continuing to talk to each other, eventually we will all speak the same language, when we say “Java 8”. Please bear with us as the saga continues …


PS: Another pet peeve

I am a type system enthusiast, mostly, because I like how type checkers can completely eliminate entire categories of bugs from your programs. I like to give a guarantee that no code that is accepted by the compiler will ever fail at runtime with an exception like attempting to invoke a method that is not present on the receiver, or class cast exceptions in source code that doesn’t mention any class cast. Type inference is the tool that alleviates the verbosity of explicitly typed programs, while at the same time maintaining the same guarantees about type safety.

Unfortunately, there is a class of Java programs for which such guarantees can not be given: if a program uses raw types, the compiler needs to generate lots of checkcast instructions, to make the code acceptable for the JVM. Each of these instructions can cause the program to blow up at runtime in totally unsuspicious locations.

There are situations where javac silently assumes that a raw type List is a subtype of its parameterized form List<String>. This is wrong. Still I cannot just ignore this problem, because lots of “bugs” are reported against ecj, based on the observation that javac and ecj accept different programs, where in many cases the difference concerns the handling of raw types during type inference.

Economically speaking, investigating in the subtleties of how Java 8 compilers handle raw types is a huge waste of efforts. Any one reading this: if you want to do me a favor, and thus help me to focus on the most relevant aspects of compiler development, please clean up your code. If you keep your code private, nobody will suffer except from yourself, but please, before posting a bug report against JDT, if your code example contains raw types, think thrice before submitting the report. Adding proper type arguments will certainly improve the quality of your code. Likely, after that exercise also ecj will be a lot happier with your code and give you correct answers.

Do I need to repeat that raw types were a workaround for migrating towards Java 5? … that raw types were discouraged starting from day 1 of Java 5? If that doesn’t convince you, search on StackOverflow for questions mentioning raw types and type inference, and you will see that by the use of raw types you are basically disabling much of the power of type inference. Let’s please shed the legacy of raw types.

Written by Stephan Herrmann

April 2, 2017 at 21:43

Posted in Eclipse, Uncategorized

Tagged with , ,

Eclipse Neon.2 is on Maven Central

with 6 comments

It’s done, finally!

Bidding farewell to my pet peeve

In my job at GK Software I have the pleasure of developing technology based on Eclipse. But those colleagues consuming my technology work on software that has no direct connection to Eclipse nor OSGi. Their build technology of choice is Maven (without tycho that is). So whenever their build touches my technology we are facing a “challenge”. It doesn’t make a big difference if they are just invoking a code generator built using Xtext etc or whether some Eclipse technology should actually be included in their application runtime.

Among many troubles, I recall one situation that really opened my eyes: one particular build had been running successfully for some time, until one day it was fubar. One Eclipse artifact could no longer be resolved. Followed long nights of searching why that artifact may have disappeared, but we reassured ourselves, nothing had disappeared. Quite to the contrary somewhere on the wide internet (Maven Central to be precise) a new artifact had appeared. So what? Well, that artifact was the same that we also had on our internal servers. Well, if it’s the same, what’s the buzz? It turned out it had a one-char difference in its version: instead of 1.2.3.v20140815 its version was 1.2.3-v20140815. Yes take a close look, there is a difference. Bottom line, with both almost-identical versions available, Maven couldn’t figure out what to do, maybe each was considered as worse than the other, to the effect that Maven simply failed to use either. Go figure.

More stories like this and I realized that relying on Eclipse artifacts in Maven builds was always at the mercy of some volunteers, who typically don’t have a long-term relationship to Eclipse, who filled in a major gap by uploading individual Eclipse artifacts to Maven Central (thanks to you volunteers, please don’t take it personally: I’m happy that your work is no longer needed). Anybody who has ever studied the differences between Maven and OSGi (wrt dependencies and building that is) will immediately see that there are many possible ways to represent Eclipse artifacts (OSGi bundles) in a Maven pom. The resulting “diversity” was one of my pet peeves in my job.

At this point I decided to be the next volunteer who would screw up other people’s builds who would collaborate with the powers that be at Eclipse.org to produce the official uploads to Maven Central.

As of today, I can report that this dream has become reality, all relevant artifacts of Neon.2 that are produced by the Eclipse Project, are now “officially” available from Maven Central.

Bridging between universes

I should like to report some details of how our artifacts are mapped into the Maven world:

The main tool in this endeavour is the CBI aggregator, a model based tool for transforming p2 repositories in various ways. One of its capabilities is to create a Maven repository (a dual use repo actually, but the p2 side of this is immaterial to this story). That tool does a great job of extracting meta data from the p2 repo in order to create “meaningful” pom files, the key feature being: it copies all dependency information, which is originally authored in MANIFEST.MF, into corresponding declarations in the pom file.

Still a few things had to be settled, either by improving the tool, by fine tuning the input to the tool, or by some steps of post-processing the resulting Maven repo.

  • Group IDs
    While OSGi artifacts only have a single qualified Bundle-SymbolicName, Maven requires a two-part name: groupId x artifactId. It was easy to agree on using the full symbolic name for the artifactId, but what should the groups be? We settled on these three groups for the Eclipse Project:

    • org.eclipse.platform
    • org.eclipse.jdt
    • org.eclipse.pde
  • Version numbers
    In Maven land, release versions have three segments, in OSGi we maintain a forth segment (qualifier) also for releases. To play by Maven rules, we decided to use three-part versions for our uploads to Maven Central. This emphasizes the strategy to only publish releases, for which the first three parts of the version are required to be unique.
  • 3rd party dependencies
    All non-Eclipse artifacts that we depend on should be referenced by their proper coordinates in Maven land. By default, the CBI aggregator assigns all artifacts to the synthetic group p2.osgi.bundle, but if s.o. depends on p2.osgi.bundle:org.junit this doesn’t make much sense. In particular, it must be avoided that projects consuming Eclipse artifacts will get the same 3rd party library under two different names (perhaps in different versions?). We identified 16 such libraries, and their proper coordinates.
  • Source artifacts
    Eclipse plug-ins have their source code in corresponding .source plug-ins. Maven has a similar convention, just using a “classifier” instead of appending to the artifact name. In Maven we conform to their convention, so that tools like m2e can correctly pick up the source code from any dependencies.
  • Other meta data
    Followed a hunt for project url, scm coordinates, artifact descriptions and related data. Much of this could be retrieved from our MANIFEST.MF files, some information is currently mapped using a static, manually maintained mapping. Other information like licences and organization are fully static during this process. In the end all was approved by the validation on OSSRH servers.

If you want to browse the resulting wealth, you may start at

Everything with fully qualified artifact names in these groups (and date of 2017-01-07 or newer) should be from the new, “official” upload.

This is just the beginning

The bug on which all this has been booked is Bug 484004: Start publishing Eclipse platform artifacts to Maven central. See the word “Start”?

To follow-up tasks are already on the board:

(1) Migrate all the various scripts, tools, and models to the proper git repo of our releng project. At the end of the day, this process of transformation and upload should become a routine operation to be invoked by our favourite build meisters.

(2) Fix any quirks in the generated pom files. E.g., we already know that the process did not handle fragments in an optimal way. As a result, consuming SWT from the new upload is not straight forward.

Both issues should be handled in or off bug 510072, in the hope, that when we publish Neon.3 the new, “official” Maven coordinates of Eclipse artifacts will be even fit all all real world use. So: please test and report in the bug any problems you might find.

(3) I was careful to say “Eclipse Project”. We don’t yet have the magic wand to apply this to literally all artifacts produced in the Eclipse community. Perhaps s.o. will volunteer to apply the approach to everything from the Simultaneous Release? If we can publish 300+ artifacts, we can also publish 7000+, can’t we? 🙂

happy building!

Written by Stephan Herrmann

January 9, 2017 at 23:21

Posted in Eclipse, Uncategorized

Tagged with , , , ,

Runtime Specialization – At Last

with one comment

Between a rock and a hard place

Not long ago, I had to pull Object Teams out of the Eclipse simultaneous release train. Reason: the long standing issue of using BCEL for bytecode weaving, for which no Java 8 compatible version has yet been released. With the Eclipse platform moving to Java 8, this had escalated to a true blocker. During the last weeks I investigated two options in parallel:

  • Upgrade BCEL to a release candidate of the upcoming 6.0 release.
  • Finalize the alternative weaver for OT/J, which is based on ASM, and thus already capable of handling Java 8 byte codes

I soon found out that even BCEL 6.0 will not be a full solution, because it still has no real support for creating StackMapTable attributes for newly generated bytecode, which however is strictly mandatory for running on a JVM 8.

For that reason I then focussed on the OTDRE, the Object Teams Dynamic Runtime Environment. This has been announced long ago, and after all, I promised to show a sneak preview of this feature in my presentation at EclipseCon Europe:

Runtime Specialization
Java has never been so dynamic before

Success at last

Today I can report success in two regards:

  • The Object Teams Development Tooling, which itself is a complex OT/J application, can (mostly) run on the new runtime!
  • I created a first demo example that shows the new capability of runtime weaving in action – it works! 🙂

This is a major milestone! Running OTDT on top of OTDRE is a real stress test for that new component – once again I realize that dog-fooding an entire IDE on its own technology is quite an exciting exercise. While a few exceptions need to be ironed out before the Neon release, I’m confident now, that we’re finally and really on the home stretch of this effort.

But OTDRE is not just a replacement for the traditional runtime, it is also way cooler, as the second success story shows: it is indeed possible now, to throw new teams and roles into a running application. If the application has been generically prepared for this task, the new teams can be automatically activated and immediately start adapting the running application – no specific preplanning needed. With this we are able to achieve a level of dynamism that typically is only possible with dynamic languages. And all this without any compromise to static typing and analysability. This is probably too cool to be allowed.

And after all the hard work on Java 8, also OT/J can finally fully leverage the new version, not only in theory, but also in bytecode.

Less than one week to finalize the presentation. You can be sure this will be a fresh story. Join me on Wednesday, Nov 4,  in Ludwigsburg:

Runtime Specialization - Java has never been so dynamic before -- Session at EclipseCon Europe 2015

PS: The “traditional” Object Teams Runtime Environment isn’t dead, yet. I really want to keep it as an option, because both variants (OTRE / OTDRE) have quite different characteristics, and after all this component has matured over more than 10 years. But with one option already (mostly) working, I can probably wait until a proper release of BCEL 6.0, and still have it back in game before the Neon release.

Written by Stephan Herrmann

October 27, 2015 at 23:05

Object Teams now has Lambdas (and more)

leave a comment »

In a previous post I announced that the Luna version of Object Teams will incorporate support for Java 8.

Now that the big rush towards March 18 is over, I took stock how Object Teams fits into the Luna stream, which finally contains all the bits for Java 8 support. The result:

We have our first builds with full support for OT/J on Java 8!

This means, you can now use lambda expressions and more even in OT/J programs, here’s a witness:

Lambda In Parameter Mapping

While this program may not make a lot of sense, it demonstrates a funny piece of syntax. Here’s what you see:

  • The regular class PersonCollection offers a method restrict that takes an argument of a functional type (Predicate)
  • Role Club.Members is played by PersonCollection and makes the base method available via a callout binding (lines 19 f)
  • The role wants to pass just an int value, where the base method expects a Predicate.
  • This incompatibility is bridged by a parameter mapping that maps a lambda expression to the expected parameter (line 20).

This looks funny, because two different uses of the token -> meet on a single line: the first -> is part of the lambda expression, separating the parameter from the body, the second -> is part of the parameter mapping, mapping the expression on the left to the parameter on the right.

Were I to chose the syntax in a green field situation, I would certainly not use the same token for such different purposes. But since we’re basically merging two independent language extensions I had no choice. After some worries that the combined language might not be parseable I’m much relieved to see it actually working!

A very similar situation actually exists concerning type annotations. More on that in a future post.

Remaining steps towards Luna

Merging the entire BETA_JAVA8 code base from JDT into Object Teams is done, but a few regressions (< 30 out of 84,000+ tests) remain from that exercise. But compared to the original JDT efforts for Java 8 this truly is peanuts 🙂

We only have one hurdle still to jump: our bytecode weaver is based on BCEL, for which no update for Java 8 is in sight. This means: we cannot weave into class files that use the Java 8 bytecode version, bummer!

So the real interesting question will be: can we get the long announced alternate weaver using ASM up to speed in time for Luna? I think we can, but it won’t hurt if you wish me luck in this endeavour.

And yes: one day I should start writing meaningful examples using Java 8 features in OT/J code…

Written by Stephan Herrmann

March 30, 2014 at 23:44

Posted in Eclipse, Object Teams, OTDT, OTJLD

Tagged with ,

How to kick the fly-shuttle

leave a comment »

This post was originally written on August 6th, 2013, but got lost when moving this blog. Luckily I could recover it from backup.

Software composition by “weaving”

Programming languages in the wider field of Aspect Oriented Software Development introduce some kinds of composition concepts, that cannot directly be mapped to pure Java. We got used to speaking of “weaving” as a mechanism for preparing classes in these languages for execution on the JVM, whereas a true aspect oriented virtual machine would natively perform this composition transparently and more efficiently. But with the dominance of the JVM it will be “weaving” for a long time to come.

fly-shuttle

Back in the early days, AOSD languages performed some static composition on source or class files. When OT/J entered the stage, first frameworks emerged that allowed us to hook into the class loading process, so we could do the weaving dynamically on-the-fly, without ever storing the woven classes on disk. Work is (still) in the pipe-line to support even runtime-(re)weaving.

One theme in all the development of Object Teams is: first solve the dynamic case, that’s where the challenges lie, and leave more static cases for later as an optimization. “Later” has come now.

How (not) to hook into class loading?

A little while into the development of OT/J, Java 5 was published introducing the JPLIS API and the concept of a -javaagent. Today many instrumentation tools use this facility and OT/J was among the first on this boat. So for standalone Java applications the problem is solved.

Later, while developing Eclipse plug-ins, we noticed that it wouldn’t be overly cool if you have to invoke Eclipse with a -javaagent. Not only is this very inconvenient for installation, it is also problematic (e.g., for performance) to feed all classes of Eclipse through our weaver. We were lucky again, and the Equinox Adaptor Hooks emerged right when we needed them. With these hooks we were able to integrate OT/J in a way that enables all the on-the-fly flexibility we want while at the same time it lets us control the process so that the flexibility cannot be abused. So in Equinox (or better: OT/Equinox) the problem is (more than) solved.

The fact that Eclipse Luna will no longer support these hooks was slightly alarming news for Object Teams, but that problem is already essentially solved – details to be reported soon.

I’ve heard about successful experiments running OT/J code in tomcat, but, hey, the list of application servers/containers is long, and if we need specifically crafted integration for each environment, it’s kind of difficult to argue that OT/J runs “everywhere”. Is there a generic why to kick the fly-shuttle?

fly-shuttle-2

Dynamic weaving is cool, but not having a fallback is uncool, so I finally developed the static scenario as a special case of the dynamic one: a build-time weaver for OT/J.

The little new tool is ridiculously simple: we have a weaver capable of load-time weaving. All I had to do is: create a little utility that would invoke this weaver ahead-of-time, i.e., during building.

While I’m not a big fan of Maven, in my day job Maven is a must, so I bit the bullet used the opportunity to learn a bit about development of Maven plug-ins. I was amazed to find lots of tutorials still suggesting to use API that are deprecated in Maven 3 and more surprised to read forum threads suggesting to still use that deprecated API, partly to be compatible with Maven 2 (OK?) and partly because deprecation doesn’t always mean there is an alternative (OOPS?). It’s also interesting to see, that much fundamental, up-to-date information is not available from a central source but only from tribal knowledge in hundreds of forums, blogs and whatnots. — Enough whining, on to the new stuff.

For simple projects, all you need now is a pom that looks about like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 
    <modelVersion>4.0.0</modelVersion>
 
    <parent>
        <groupId>org.eclipse.objectteams</groupId>
        <artifactId>objectteams-parent-pom</artifactId>
        <version>2.2.0</version>
    </parent>
 
    <artifactId>OTStopwatch_Built-Time_Weaver_Example</artifactId>
    <version>0.0.1-SNAPSHOT</version>
 
    <properties>
        <tycho.version>0.18.0</tycho.version>
        <otj.version>2.2.0</otj.version>
    </properties>
 
    <build>
        <plugins>
            <plugin>
                <groupId>org.eclipse.objectteams</groupId>
                <artifactId>objectteams-weaver-maven-plugin</artifactId>
                <version>0.8.0-SNAPSHOT</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>weave</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <teamClasses>
                        <teamClass>org.eclipse.objectteams.example.stopwatch.WatchUI</teamClass>
                        <teamClass>org.eclipse.objectteams.example.stopwatch.WatchUIAnalog</teamClass>
                    </teamClasses>
                </configuration>
            </plugin>
        </plugins>
    </build>
 
    <repositories>
        <repository>
            <id>ObjectTeamsRepository</id>
            <name>Object Teams Repository</name>
            <url>http://download.eclipse.org/objectteams/maven/3/repository</url>
        </repository>
    </repositories>
 
   <dependencies>
       <dependency>
           <groupId>org.eclipse.objectteams</groupId>
           <artifactId>objectteams-runtime</artifactId>
           <version>${otj.version}</version>
       </dependency>
   </dependencies>
</project>
 

Here we enable OT/J compilation via the parent pom, specify a repo where to get the OT artifacts, a dependency and a few versions, nothing sophisticated.

What’s new is the objectteams-weaver-maven-plugin, which is minimally configured just by specifying a list of team classes that should be woven. Oops, that’s already it.

By integrating this into your build, you’ll produce class files that can be directly executed “everywhere”, i.e., without the OT/J load-time weaver, and thus: without fiddling with the class loading process. One caveat remains: naturally, by producing the woven class files offline, it will be your responsibility to correctly feed the correct versions of class files into the classpath in the desired order. If you can’t use the load-time weaver, it won’t be able to help keep things simple.

More configuration options are documented on the plug-in’s site and more explanation using our favorite examples can be found in the wiki.

Here you have it: static weaving falls of naturally if you already have dynamic weaving. And the resulting class files can truly be run “everywhere”.

I’d be curious to hear of the first OT/J apps on Android (though personally I’d prefer ubuntu).

Written by Stephan Herrmann

January 28, 2014 at 18:16

Posted in Eclipse, Object Teams, OTEquinox

Tagged with , , ,

Object Teams in the times of Eclipse Luna and Java 8

with one comment

With Eclipse Luna approaching milestone 5 it’s time to write a bit about what this year will bring for Object Teams.

Overhaul of the OT/Equinox weaver

The Luna cycle started with a surprise (for me): Equinox no longer provides the Adaptor Hooks, which allowed OT/Equinox to hook into the OSGi class loading process and perform its bytecode weaving. Outch!

On the other hand, OSGi meanwhile has a standard WeavingHook for doing this kind of stuff. With some efforts (and lots of help from Tom W. – no help from p2, though) I managed to migrate OT/Equinox to this new standard. At the end of the day, this will bring better encapsulation and hopefully better launch performance, too. Details to be posted some time later.

Lift OT/J to be based on Java 8

Some may have noticed that most my Eclipse-time is currently being spent in helping JDT to get ready for Java 8. And sure this is an exciting endeavour to be part of!

For Object Teams the cool part about this is: I’ve seen the changes for Java 8 happen in JDT, which greatly helps to adopt these changes for OT/J. Once in a while this even reveals a bug in JDT before it ever surfaced 🙂

The integration of OT/J with Java 8 still has some regressions, but my plan is to have this at good “milestone quality” when Java 8 is released in March and to get it to full release quality for Luna GA in June.

Question: does it actually make sense to combine lambdas and type annotations with roles and teams? I strongly believe it does, because these are language improvements in entirely different categories:

  • Lambda expressions help to implement algorithms in a more concise and also more modular way – this lets you think differently about functions.
  • Type annotations help enhance safety when used together with better program analysis (like, e.g., enhanced null analysis) – these let you think differently about types.
  • Roles and teams help improve the program structure at a larger scale – these let you think differently about classes and objects.

So, if each of these additions makes sense, then combining them all in one language will certainly yield a very powerful language. Some examples of combination to follow as we approach the release.

ObjectTeams.org is down, long live ObjectTeams.org

Not all material from the 10+ years of history of Object Teams has moved to Eclipse.org. Notably the language definition (OTJLD) and scientific publications are still hosted on objectteams.org. Until recently, my former University kindly provided the host for publishing these research results. Now that server has gone off service and for a while objectteams.org was dead — but as of today the relevant content is back online – sorry for the disruption. And, btw, this is also the reason why this blog has changed its URL.

Please wish me luck for the work ahead, both on JDT and Object Teams 🙂

Written by Stephan Herrmann

January 21, 2014 at 22:37

Book chapter published: Confined Roles and Decapsulation in Object Teams — Contradiction or Synergy?

leave a comment »

I strongly believe that for perfect modularity, encapsulation in plain Java is both too weak and too strong. This is the fundamental assumption behind a book chapter that has just been published by Springer.

The book is:
Aliasing in Object-Oriented Programming. Types, Analysis and Verification
My chapter is:
Confined Roles and Decapsulation in Object Teams — Contradiction or Synergy?

The concepts in this chapter relate back to the academic part of my career, but all of my more pragmatic tasks since those days indicate that we are still very far away from optimal modularity, and both mistakes are found in real world software: to permit access too openly and to restrict access too strictly. More often than not, it’s the same software exhibiting both mistakes at once.

For the reader unfamiliar with the notions of alias control etc., let me borrow from the introduction of the book:

Aliasing occurs when two or more references to an object exist within the object graph of a running program. Although aliasing is essential in object-oriented programming as it allows programmers to implement designs involving sharing, it is problematic because its presence makes it difficult to reason about the object at the end of an alias—via an alias, an object’s state can change underfoot.

978-3-642-36946-9

Aliasing, by the way, is one of the reasons, why analysis for @Nullable fields is a thorny issue. If alias control could be applied to @Nullable fields in Java, much better static analysis would be possible.


How is encapsulation in Java too weak?

Java only allows to protect code, not objects

This manifests at two levels:

Member access across instances

In a previous post I mentioned that the strongest encapsulation in Java – using the private modifier – doesn’t help to protect a person’s wallet against access from any other person. This is a legacy from the pre-object-oriented days of structured programming. In terms of encapsulation, a Java class is a module utterly unaware of the concept of instantiation. This defect is even more frustrating as better precedents (think of Eiffel) have been set before the inception of Java.

A type system that is aware of instances, not just classes, is a prerequisite for any kind of alias control.

Object access meets polymorphism

Assume you declare a class as private because you want to keep a particular thing absolutely local to the current module. Does such a declaration provide sufficient protection? No. That private class may just extend another – public – class or implement a public interface. By using polymorphism (an invisible type widening suffices) an instance of the private class can still be passed around in the entire program – by its public super type. As you can see, applying private at the class level, doesn’t make any objects private, only this particular class is. Since every class extends at least Object there is no way to confine an object to a given module; by widening all objects are always visible to all parts of the program. Put dynamic binding of method calls into the mix, and all kinds of “interesting” effects can be “achieved” on an object, whose class is “secured” by the private keyword.

The type system of OT/J supports instance-based protection.

Java’s deficiencies outlined above are overcome in OT/J by two major concepts:

Dependent types
Any role class is a property of the enclosing team instance. The type system allows reasoning about how a role is owned by this enclosing team instance. (read the spec: 1.2.2)
Confined roles
The possible leak by widening can be prevented by sub-classing a predefined role class Confined which does not extend Object. (read the spec: 7.2)

For details of the type system, why it is suitable for mending the given problems, and why it doesn’t hurt in day-to-day programming, I have to refer you to the book chapter.


How is encapsulation in Java too strict?

If you are a developer with a protective attitude towards “your” code, you will make a lot of things private. Good for you, you’ve created (relatively) well encapsulated software.
But when someone else is trying to make use of “your” code (re-use) in a slightly unanticipated setting (calling for unanticipated adaptations), guess what: s/he’ll curse you for your protective attitude.
Have you ever tried to re-use an existing class and failed, because some **** detail was private and there was simply no way to access or override that particular piece? When you’ve been in this situation before, you’ll know there are basically 2 answers:

  1. Give up, simply don’t use that (overly?) protective class and recode the thing (which more often than not causes a ripple effect: want to copy one method, end up copying 5 or more classes). Object-orientation is strong on re-use, heh?
  2. Use brute force and don’t tell anybody (tools that come in handy are: binary editing the class file, or calling Method.setAccessible(true)). I’m not quite sure why I keep thinking of Core Wars as I write this 🙂 .

OT/J opens doors for negotiation, rather than arms race & battle

The Object Teams solution rests on two pillars:

  1. Give a name to the act of disregarding an encapsulation boundary: decapsulation. Provide the technical means to punch tiny little holes into the armor of a class / an object. Make this explicit so you can talk and reason about the extent of decapsulation. (read the spec: 2.1.2(c))
  2. Separate technical possibility from questions of what is / should / shouln’t be allowed. Don’t let technology dictate rules but make it possible to formulate and enforce such rules on top of the existing technology.

Knowing that this topic is controversial I leave at this for now (a previous discussion in this field can be found here).


Putting it all together

  1. If you want to protect your objects, do so using concepts that are stronger than Java’s “private”.
  2. Using decapsulation where needed fosters effective re-use without the need for “speculative API”, i.e., making things public “just in case”.
  3. Dependent types and confined roles are a pragmatic entry into the realm of programs that can be statically analysed, and strong guarantees be given by such analysis.
  4. Don’t let technology dictate the rules, we need to put the relevant stakeholders back into focus: Module providers, application developers and end users have different views. Technology should just empower each of them to get their job done, and facilitate transparency and analyzability where desired.

Some of this may be surprising, some may sound like a purely academic exercise, but I’m convinced that the above ingredients as supported in OT/J support the development of complex modular systems, with an unprecedented low effective coupling.

FYI, here’re the section headings of my chapter:

  1. Many Faces of Modularity
  2. Confined Roles
    1. Safe Polymorphism
    2. From Confined Types to Confined Roles
    3. Adding Basic Flexibility to Confined Roles
  3. Non-hierarchical Structures
    1. Role Playing
    2. Translation Polymorphism
  4. Separate Worlds, Yet Connected
    1. Layered Designs
    2. Improving Encapsulation by Means of Decapsulation
    3. Zero Reference Roles
    4. Connecting Architecture Levels
  5. Instances and Dynamism
  6. Practical Experience
    1. Initial Comparative Study
    2. Application in Tool Smithing
  7. Related Work
    1. Nesting, Virtual Classes, Dependent Types
    2. Multi-dimensional Separation of Concerns
    3. Modules for Alias Control
  8. Conclusion

Written by Stephan Herrmann

March 30, 2013 at 22:11