Retrospective of an Old Man (2)

(continued from here).

Today I want to speak about four engineers who have influenced Eclipse JDT in the past: Srikanth, Markus Keller, Till Brychcy and Yours Truly. All of them have left the team over the course of the years. None has left because they didn’t love Eclipse. It would come closer to the truth to say, they left because they loved Eclipse too much. I am not authorized to publicly speak about the personal motives of these people. For that reason I will generalize as if all four situations were identical. Obviously this isn’t true, but still I sense a common gist, some overarching topics that might still be relevant today.

What is said below is meant as observations about these four engineers only. Whatever quality I ascribe to them, may well be shared by others not mentioned here. It’s not about drawing boundaries, but about perceived commonality.

Qualification

Every healthy community has members with a very focused area of expertise, in which their knowledge is thorough and deep. Other members may have a wide range of interest and involvement, connecting the dots between different focus areas. Some members excel by way of sharp analytical capabilities, others contribute the experience from years of working on a component. The personae in this story drew their strength from combining (at different degrees) all four qualifications: knowledge both deep and wide, understanding both analytically sharp and founded on a body of experience.

Responsibility

While every contributor takes responsibility for their contribution, some people feel responsible for an entire component or even product. In this text I don’t speak of responsibility from a managerial perspective, but of a code-centric point of view that cares about the code in all aspects of quality.

In one of the most complex endeavors that I got involved in, one of our four engineers mentioned in retrospect: “failure was not an option“. Feeling responsible for a component implies to closely track the bugzilla inbox, and to make sure that every incoming bug report is handled with due attention. It also implies to make sure that every proposed code change is of highest possible quality, not only in terms of functionality but also as being in harmony with existing design, to prevent architectural decay. Such responsibility finally implies to care about completeness of functionality. In particular when a new Java version changes the game, JDT must support users in working in that new environment. As an example for the latter aspect think of the conflict between unit testing and the Java Platform Module System. In this context, treating test sources differently from main sources is a must, which is why Till added this functionality to JDT (a huge project of its own).

Burn-Out

Working with the attitude described above requires a lot of energy. In a healthy community this energy flows in both directions, and everything may well be sustainable. In the cases of our four engineers, however, too much energy was burned, obstacles appeared and caused frustration. Some may get frustrated when they see code changes that are made with insufficient understanding, unaware of maintenance costs incurred. Sometimes it’s the tone of discussion that kills any pleasure of contributing. There’s also a structural conflict between meritocracy and unlimited openness of a community: Think of our engineer who wants to take responsibility, he will have to make some decisions. He feels that the rules of meritocracy grant him permission to make such decision and furthermore he is convinced that such decision is necessary to fulfill the assumed responsibility. If the community doesn’t accept such decisions, our engineer would have to fight an extra (seemingly unnecessary) battle just to get permissions for enacting a decision he deems necessary for the greater goal.

In all four cases the community failed to balance the flow of energy, to get obstacles out of the way, to empower those who feel responsible to enact that responsibility. Not all four experienced a full-blown burn-out, but that’s more a question of whether or not they pulled the plug in good time.

Two of our engineers actually had to quit their job to leave Eclipse, the other two didn’t have to ask anybody, because they never received a penny for their contributions.

Questions

Is it just normal that engineers risk burn-out? Is it something happening to everybody? Is it acceptable? Does it make a difference to the community who is it who gets frustrated? More generally speaking: is the community willing to make any difference between newbies and potential “meritocrats”? Does the community actually support the goal of excellent code quality, or is speed of change, e.g., considered more important?

In my previous post I spoke of the end of an era. Does the new era have use for engineers like these? It may not be an extinct species, but I’m afraid their number is dwindling…

Clarification

No, in this post I’m not begging for any “presents” that would lure me back into my previous position. I had my share of responsibility. Quite likely I had bitten off more then I could possibly chew, when I felt responsible not only for all of the compiler (ecj) per-se, but also to drive Oracle towards improving JLS in ways that would enable the teams behind ecj and javac to finally converge on the same language. I felt responsible at other levels, too, but I will not enumerate my merits. I made a final decision to shed all this responsibility – in good time before it would affect my health.

Advertisement

Retrospective of an Old Man

Last summer I dropped my pen concerning contributions for Eclipse JDT. I never made a public announcement about this, but half a year later I started to think: doesn’t it look weird to receive a Lifetime Achievement Award and then run off without even saying thanks for all the fish? Shouldn’t I at least try to explain what happened? I soon realized that writing a final post to balance accounts with Eclipse would neither be easy nor desirable. Hence the idea, to step back more than a couple of steps, and put my observations on the table in smaller chunks. Hopefully this will allow me to describe things calmly, perhaps there’s even an interesting conclusion to be drawn, but I’ll try to leave that to readers as much as I can.

Prelude

While I’m not yet preparing for retirement, let me illustrate the long road that led me where I am today: I have always had a strong interest in software tools, and while still in academia (during the 1990s) my first significant development task was providing a specialized development environment (for a “hybrid specification language” if you will). That environment was based on what I felt to be modern at that time: XEmacs (remember: “Emacs Makes A Computer Slow”, the root cause being: “Eight Megabytes And Continuously Swapping”). I vaguely remember a little time later I was adventurous and installed an early version of NetBeans. Even though the memory of my machine was upped (was it already 128 MB?), that encounter is remembered as surpassing the bad experience of Emacs. I never got anything done with it.

A central part of my academic activity was in programming language development in the wider area of Aspect Oriented Software Development. For pragmatical reasons (and against relevant advice by Gilad Bracha) I chose Java as the base language to be extended to become ObjectTeams/Java, later rebranded as OT/J. I owe much to two students, whose final projects (“Diplomarbeit”) was devoted to two successive iterations of the OT/J compiler. One student modified javac version 1.3 (the implementation by Martin Odersky, adopted by Sun just shortly before). It was this student who first mentioned Eclipse to me, and in fact he was using Eclipse for his work. I still have a scribbled note from July 2002 “I finally installed Eclipse” – what would have been some version 2.0.x.

One of the reasons for moving forward after the javac-based compiler was: licensing. Once it dawned on me, that Eclipse contains an open-source Java compiler with no legal restrictions regarding our modifications, I started to dream about more, not just a compiler but an entire IDE for Object Teams! As a first step, another student was assigned the task to “port” our compiler modifications from javac to ecj. Some joint debugging sessions (late in 2002?) with him where my first encounters with the code base of Eclipse JDT.

First Encounters with Eclipse

For a long period Eclipse to me was (a) a development environment I was eager to learn, and (b) a huge code base in CVS to slowly wrap our heads around and coerce into what we wanted it to be. Surely, we were overwhelmed at first, but once we had funding for our project, a nice little crowd of researchers and students, we gradually munched our way through the big pile.

I was quite excited, when in 2004 I spotted a bug in the compiler, reported it in bugzilla (only to learn, that it had already been fixed 🙂 ). It took two more years until the first relevant encounter: I had spotted another compiler bug, which was then tagged as a greatbug. This earned me my first Eclipse T-shirt (“I helped make Callisto a better place“). It’s quite washed out, but I still highly value it (and I know exactly one more person owning the same T-shirt, hi Ed 🙂 ).

Soon after, I met some of my role models in person: at ECOOP 2006 in Nantes, the Eclipse foundation held a special workshop called “eTX – Eclipse Technology Exchange“, which actually was a superb opportunity for people from academia to connect with folks at Eclipse. I specifically recall inspiring chats with Jerome Lanneluc (an author of JDT’s Java Model) and Martin Aeschlimann (JDT/UI). That’s when I learned how welcoming the Eclipse community is.

During the following years, I attended my first Eclipse summits / conferences and such. IIRC I met Philippe Mulet twice. I admired him immensely. Not only was he lead developer of JDT/Core, responsible for building much of the great stuff in the first place. Also he had just gone through the exercise of moving JDT from Java 1.4 to Java 5, a task that cannot be overestimated. Having spoken to Philippe is one of the reasons why I consider myself a member of a second generation at Eclipse: a generation that still connects to the initial era, though not having been part of it.

End of an Era

For me, no other person represents the initial era of Eclipse as much as Dani did. That era has come to an end (silence).

Still a few people from the first generation are around.

Tom Watson is as firm as a rock in maintaining Equinox, with no sign of fatigue. I think he really is up for an award.

Olivier Thomann (first commit 2002) still responsibly handles issues in a few weird areas of JDT (notably: computation of StackMaps, and unicode handling).

John Arthorne has been seen occasionally. He was the one who long, long time ago explained to me the joke behind package org.eclipse.core.internal.watson (it’s elementary).

What was it like to join the community?

It wasn’t before 2010 that I became a committer for JDT/Core. I was the first committer for JDT who was not paid by IBM. I was immensely flattered by the offer. So getting into the inner circle took time: 6 years from first bug report until committer status, while all the time I was more or less actively hacking on our fork of JDT. This is to say: I was engaged with the code all the time. Did I expect things to move faster? No.

Even after getting committer status, for several years mutual reviews of patches among the team were the norm. The leads (first Olivier, then Srikanth) were quite strict in this. Admittedly, I had to get used to that – my patches waiting for reviews, my own development time split between the “real work” and “boring” reviews. In retrospect the safety net of peer reviews was a life saver – plus of course a great opportunity to improve my coding and communication skills. Let me emphasize: this was one committer reviewing the patches of another committer.

I had much respect for the code base I worked with. While working in academia, I had never seen such a big and complex code base before. And yet, there was no part that could not be learned, as all the code showed a clear and principled design. I don’t know how big the impact of Eric Gamma on details of the code was, but clearly the code spoke with the same clarity as the GoF book on design patterns.

As such, I soon learned a fundamental principle for newcomers: “Monkey see, monkey do“. I appreciated this principle because it held the promise that my own code might share the same high quality as the examples I found out there. In later days, I heard a similar attitude framed as “When in Rome, do as Romans do“.

My perspective on JDT has always been determined by entering through the compiler door. For myself this worked out extremely well, since nothing helps you understand Java in more depth and detail, than fixing compiler bugs. And understanding Java better than average I consider a prerequisite for successfully working on JDT.

to be continued

Interfacing null-safe code with legacy code

When you adopt null annotations like these, your ultimate hope is that the compiler will tell you about every possible NullPointerException (NPE) in your program (except for tricks like reflection or bytecode weaving etc.). Hallelujah.

Unfortunately, most of us use libraries which don’t have the blessing of annotation based null analysis, simply because those are not annotated appropriately (neither in source nor using external annotations). Let’s for now call such code: “legacy”.

In this post I will walk through the options to warn you about the risks incurred by legacy code. The general theme will be:

Can we assert that no NPE will happen in null-checked code?

I.e., if your code consistently uses null annotations, and has passed analysis without warnings, can we be sure that NPEs can only ever be thrown in the legacy part of the code? (NPEs inside legacy code are still to be expected, there’s nothing we can change about that).

Using existing Eclipse versions, one category of problems would still go undetected whereby null-checked code could still throw NPE. This has been recently fixed .

Simple data flows

Let’s start with simple data flows, e.g., when your program obtains a value from legacy code, like this:

You shouldn’t be surprised, the javadoc even says: “The method returns null if the property is not found.” While the compiler doesn’t read javadoc, it can recognize that a value with unspecified nullness flows into a variable with a non-null type. Hence the warning:

Null type safety (type annotations): The expression of type ‘String’ needs unchecked conversion to conform to ‘@NonNull String’

As we can see, the compiler warned us, so we are urged to fix the problem in our code. Conversely, if we pass any value into a legacy API, all bad that can happen would happen inside legacy code, so nothing to be done for our mentioned goal.

The underlying rule is: legacy values can be safely assigned to nullable variables, but not to non-null variables (example Properties.getProperty()). On the other hand, any value can be assigned to a legacy variable (or method argument).

Put differently: values flowing from null-checked to legacy pose no problems, whereas values flowing the opposite direction must be assumed to be nullable, to avoid problems in null-checked code.

Enter generics

Here be dragons.

As a minimum requirement we now need null annotations with target TYPE_USE (“type annotations”), but we have this since 2014. Good.

Here we obtain a List<String> value from a Legacy class, where indeed the list names is non-null (as can be seen by successful output from names.size()). Still things are going south in our code, because the list contained an unexpected null element.

To protect us from this problem, I marked the entire class as @NonNullByDefault, which causes the type of the variable names to become List<@NonNull String>. Now the compiler can again warn us about an unsafe assignment:

Null type safety (type annotations): The expression of type ‘List<String>’ needs unchecked conversion to conform to ‘List<@NonNull String>’

This captures the situation, where a null value is passed from legacy to null-checked code, which is wrapped in a non-null container value (the list).

Here’s a tricky question:

Is it safe to pass a null-checked value of a parameterized type into legacy code?

In the case of simple values, we saw no problem, but the following example tells us otherwise once generics are involved:

Again we have a list of type List<@NonNull String>, so dereferencing values obtained from that list should never throw NPE. Unfortunately, the legacy method printNames() succeeded to break our contract by inserting null into the list, resulting in yet another NPE thrown in null-checked code.

To describe this situation it helps to draw boundaries not only between null-checked and legacy code, but also to draw a boundary around the null-checked value of parameterized type List<@NonNull String>. That boundary is breached when we pass this value into legacy code, because that code will only see List<String> and happily invoke add(null).

This is were I recently invented a new diagnostic message:

Unsafe null type conversion (type annotations): The value of type ‘List<@NonNull String>’ is made accessible using the less-annotated type ‘List<String>’

By passing names into legacy code, we enable a hidden data flow in the opposite direction. In the general case, this introduces the risk of NPE in otherwise null-checked code. Always?

Wildcards

Java would be a much simpler language without wildcards, but a closer look reveals that wildcards actually don’t only help for type safety but also for null-safety. How so?

If the legacy method were written using a wildcard, it would not be (easily) possible to sneak in a null value, here are two attempts:

The first attempt is an outright Java type error. The second triggers a warning from Eclipse, despite the lack of null annotations:

Null type mismatch (type annotations): ‘null’ is not compatible to the free type variable ‘?’

Of course, compiling the legacy class without null-checking would still bypass our detection, but chances are already better.

If we add an upper bound to the wildcard, like in List<? extends CharSequence>, not much is changed. A lower bound, however, is an invitation for the legacy code to insert null at whim: List<? super String> will cause names.add() to accept any String, including the null value. That’s why Eclipse will also complain against lower bounded wildcards:

Unsafe null type conversion (type annotations): The value of type ‘List<@NonNull String>’ is made accessible using the less-annotated type ‘List<? super String>’

Comparing to raw types

It has been suggested to treat legacy (not null-annotated) types like raw types. Both are types with a part of the contract ignored, thereby causing risks for parts of the program that still rely on the contract.

Interestingly, raw types are more permissive in the parameterized-to-raw conversion. We are generally not protected against legacy code inserting an Integer into a List<String> when passed as a raw List.

More interestingly, using a raw type as a type argument produces an outright Java type error, so my final attempt at hacking the type system failed:

Summary

We have seen several kinds of data flow with different risks:

• Simple values flowing checked-to-legacy don’t cause any specific headache
• Simple values flowing legacy-to-checked should be treated as nullable to avoid bad surprises. This is checked.
• Values of parameterized type flowing legacy-to-checked must be handled with care at the receiving side. This is checked.
• Values of parameterized type flowing checked-to-legacy add more risks, depending on:
  • nullness of the type argument (@Nullable type argument has no risk)
  • presence of wildcards, unbounded or lower-bounded.

Eclipse can detect all mentioned situations that would cause NPE to be thrown from null-checked code – the capstone to be released with Eclipse 2020-03, i.e., coming soon …

Oracle made me a Stackoverflow Guru

Just today Oracle helped me to become a “Guru” on Stackoverflow! How did they do it? By doing nothing.

In former times, I was periodically enraged, when Oracle didn’t pay attention to the feedback I was giving them during my work on ecj (the Eclipse Compiler for Java) – at least not the attention that I had hoped for (to be fair: there was a lot of good communication, too). At those times I had still hoped I could help make Java a language that is completely and unambiguously defined by specifications. Meanwhile I recognized that Java is at least three languages: the language defined by JLS etc., the language implemented by javac, and the language implemented by ecj (and no chance to make ecj to conform to both others). I realized that we were not done with Java 8 even 3 years after its release. Three more years later it’s still much the same.

So let’s move on, haven’t things improved in subsequent versions of Java? One of the key new rules in Java 9 is, that

“If [a qualified package name] does not name a package that is uniquely visible to the current module (§7.4.3), then a compile-time error occurs”.

Simple and unambiguous. That’s what compilers have to check.

Except: javac doesn’t check for uniqueness if one of the modules involved is the “unnamed module”.

In 2018 there was some confusion about this, and during discussion on stackoverflow I raised this issue to the jigsaw-dev mailing list. A bug was raised against javac, confirmed to be a bug by spec lead Alex Buckley. I summarized the situation in my answer on stackoverflow.

This bug could have been easily fixed in javac version 12, but wasn’t. Meanwhile upvotes on my answer on stackoverflow started coming in. The same for Java 13. The same for Java 14. And yet no visible activity on the javac bug. You need ecj to find if your program violates this rule of JLS.

Today the 40th upvote earned me the “Guru” tag on stackoverflow.

So, please Oracle, keep that bug unresolved, it will earn me a lot of reputation for a bright future – by doing: nothing 🙂

Edit: In 2020/04, Oracle dropped any plans to resolve the issue, meanwhile my answer on stackoverflow matured into a “Great Answer” with more then 100 upvotes.

Jigsaw arriving – and now what?

The debate is over. The result of fierce discussions has been declared. Jigsaw is knocking at the door. Now the fate of Jigsaw depends on the community. Will the community embrace Jigsaw? Will we be able to leverage the new concepts to our benefit? Will we struggle with new pitfalls?

Let’s step back a second: do we really know what is Jigsaw? Do we really understand the design, its subtleties and implications?

At EclipseCon Europe 2017 I will try to shed a light on some lesser known aspects of Jigsaw. A deep dive into things I learned as a JDT committer while implementing tool support for Java 9.

In search for truth

To set the stage, we’ll first have to figure out, who or what defines Java 9 — or Jigsaw — or JPMS. This is both a question of specification vs. implementation as well as a matter of a specification spread over several documents and artifacts. Let’s try to grok Jigsaw from the legally binding sources, rather then from secondary writing and hearsay (if that is possible).

We will have to relearn some of the basic terms, like: what is a package in Java? Do packages form a hierarchy? (I will show, how both answers, yes and no, are equally right and wrong).

Jigsaw is said to do away with some “undesirable” stuff like split packages, and cyclic dependencies. Really? (Yes and no).

Encapsulation

Of course, with Jigsaw all is about encapsulation – easy to agree on, but what is it that a Java 9 module encapsulates? Only a deep understanding of the flavor of encapsulation will tell us, what exact qualities we gain from following the new discipline (it’s not about security, e.g.), and also what pains will be incurred on the path of migrating to the new model. (Hint: I will be speaking both about compile time and runtime).

Interestingly, the magic potion Jigsaw also brings its own antidote: With tightening the rules of encapsulation, also the opposite becomes necessary: they call it breaking encapsulation, for which I once coined the term “decapsulation“. I should be flattered by how close Java 9 comes to what I call “gradual encapsulation“. So, the talk can not stop at presenting just the new language concepts, also the various extra-lingual means for tuning your architecture need to be inspected through the looking glass. This is also where tools come into focus: how can JDT empower its users to use the new options without the need to dig through the sometimes cryptic syntax of command line parameters?

Loose ends

At this point we shall agree that Jigsaw is a compromise, fulfilling many of its goals and promises, while also missing some opportunities.

I will also have to say a few words about long standing API of JDT, which makes guarantees that are no longer valid in Java 9. This raises the question: what is the migration path for tools that sit on top of JDT? (There is no free lunch).

Finally, it wouldn’t be Java, if overloading wouldn’t make things worse for the newly added concepts. But: haven’t you become fond of hating, when JDT says:

The type org.foo.Bar cannot be resolved. It is indirectly referenced from required .class files

We may be seeing light at the end of the tunnel: for Jigsaw we had to revamp the guts of our compiler in ways, that could possibly help to – finally – resolve that problem. Wish me luck …

Hope to see you in Ludwigsburg, there’s much to be discussed over coffee and beer

Ludwigsburg, Germany · October 24 – 26, 2017

Several Languages Java™ 8

More than 3 years ago, on March 18, 2014, Java™ 8 was released, and on the same day Eclipse released support for this new version. I have repeatedly made the point that the Eclipse compiler for Java (ecj) as a second implementation of JLS (after javac) serves the entire Java community as a premier means for quality assurance.

Today, in April 2017, I can report that this effort is still far from complete. Still, JLS, javac and ecj do not define the exact same language. Time to take stock what these differences are about.

My own work on ecj focuses on an aspect that tries hard to remain invisible to casual users: type inference. It’s the heavy machinery behind generic methods, diamond expressions and lambdas, allowing users to omit explicit type information in many places, leaving it to the compiler to figure out the fine print.

To be honest, when we first shipped support for Java 8, I was well expecting lots of bug reports to come in, which would point out corner cases of JLS that we hadn’t implemented correctly. There was one area, that I felt particularly uneasy about: how type inference blends with overload resolution. During the Mars cycle of development Srikanth thankfully performed a major rework and clean up of this exact area.
(I can’t pass the opportunity to report what I learned from this exercise: Overloading is a huge contributor of complexity in the Java language which in (not only) my opinion doesn’t carry its own weight — not a fraction of it).

We are not done with Java 8

The fact that still 2 years after that rework we constantly receive bug reports against Java 8 type inference is unsettling in a way.

To give some numbers to it: during every 6-week milestone we fixed between 1 and 6 bugs in type inference. None of these bugs is solved in a coffee break, some compete for the title “toughest challenge I faced in my career”.

We have a total of 103 bugs explicitly marked as 1.8 inference bugs. Of these

• 17 were resolved before Java 8 GA
• 52 have been resolved in the three years since Java 8 GA
• 34 are still unresolved today.

This will likely keep me busy for at least one more year.

In the early days of Java 8 we could identify two clusters where behavioral differences between javac and ecj could be observed:

• wildcard capture
• raw types

(I’ll have a note about the latter at the end of this post).

In these areas we could comfort ourselves by pointing to known bugs in javac. We even implemented code to conditionally mimic some of these javac bugs, but frankly, establishing bug compatibility is even more difficult than truthfully implementing a specification.

Meanwhile, in the area of wildcard capture, javac has been significantly improved. Even though some of these fixes appear only in Java 9 early access builds, not in Java 8, we can observe both compilers converging, and given that the major bugs have been fixed, it is getting easier to focus on remaining corner cases. Good.

Java “8.1”

One event almost went under our radar: In February 2015 a revised version of JLS 8 was published. As part of this update, a few sentences have been added on behalf of JDK-8038747. While the spec may be hard to grok by outsiders, the goal can be explained as enabling a compiler to include more type hints from the bodies of lambda expressions that are nested inside a generic method invocation.

In fact, collecting type information from different levels in the AST was a big goal of the type inference rewrite in Java 8, but without the 2015 spec update, compilers weren’t even allowed to look into the body of a lambda, if the lambda does not specify types for its arguments.

m(a -> a.b())

What do we know about b, while types for m and a are still unknown?

Conceptually, this is immensely tricky, because generally speaking, the code in the bodies of such type-elided lambdas can mean just about anything, while the signature of the lambda is not yet known. So we were happy about the spec update, as it promised to resolve a bunch of test cases, where javac accepts programs that ecj – without the update – was not able to resolve.

Ever since, each improved version of ecj created a regression for one or more of our dear users. We debugged no end, read between the lines of JLS, but couldn’t find a solution that would satisfy users in all cases. And they kept complaining that javac had no problems with their programs, even earlier versions of ecj accepted their program, so rejecting it now must be a regression.

“Switching the alliance”

Up-to that point, I saw our main ally in the authors of JLS, Dan Smith and Alex Buckley from Oracle. In particular Dan Smith has been a tremendous help in understanding JLS 8 and analyzing where our reading of it deviated from the authors’ intention. Together we identified not only bugs in my interpretation and implementation of JLS, but also several bugs in javac.

When we iterated bugs relating to JDK-8038747 time and again, this approach was less effective, coming to no conclusion in several cases. I slowly realized, that we were reaching a level of detail that’s actually easier to figure out when working with an implementation, than at the specification level.

This is when I started to seek advice from javac developers. Again, I received very valuable help, now mostly from Maurizio Cimadamore. Previously, my perception was, that JLS is the gold standard, and any deviation or even just liberal interpretation of it is bad. During the discussion with Maurizio I learned, that in some sense javac is actually “better” than JLS, not only in accepting more type-correct programs, but also in terms of better reflecting the intention of the JSR 335 experts.

So I started to deliberately deviate from JLS, too. Instead of “blaming” javac for deviating from JLS, I now “blame” JLS for being incomplete wrt the intended semantics.

To put this effort into proportion, please consider the figure of 103 bugs mentioned above. From these, 17 bugs have a reference to JDK-8038747. Coincidentally, this is the exact same number as those great bug reports prior to Java 8 GA, that gave us the huge boost, enabling us to indeed deliver a high quality implementation right on GA. In other words, this is a huge engineering effort, and we have no idea, how close to done we are. Will we face the next round of regressions on every new release we publish?

If you work from a specification, there is a point where you feel confident that you did all that is required. Knowing that fulfilling the spec is not enough, it’s impossible to say, what is “enough”.

What is “better”?

With wildcard captures and raw types, it was easy to argue, that certain programs must be rejected by a compiler, because they are not type safe and can blow up at runtime in unexpected locations. In the area around JDK-8038747 javac tends to accept more programs than JLS, but here it would be unreasonable to expect javac to change and start rejecting these “good” programs.

Still, calling out a competition of who accepts more “good” programs would be a bad idea, too, because this would completely abandon the goal of equivalence between compilers. After compiling with one compiler, one could never be sure that another compiler would also accept the same program. The term “Java” would loose its precise meaning.

This implies, every attempt to better align ecj with javac, based on knowledge about the implementation and not based on JLS, should be seen as a temporary workaround. To resume its role of leadership, JLS must catch up with any improvements done in the implementation(s).

To comfort the reader, I should say that in all cases discussed here, there’s always a safe fallback: when inference fails to find a solution, it is always possibly to help the compiler by adding some explicit type arguments (or argument types for a lambda). More importantly, such additions, which may be required for one compiler, should never cause problems for another compiler.

Also note, that explicit type arguments are always to be preferred over type casts (which some people tend to use as a workaround): type arguments will help for effective type checking, whereas type casts bypass type checking and can blow up at runtime.

Thanks and Sorry!

I wrote this post in the desire to reach out to our users.

First: Each reproducible bug report is highly valuable; this is what drives JDT code towards higher and higher quality. By accumulating test cases from all these reports we gradually create a test suite that provides the best available safety net.

Second: I am sorry about every regression introduced by any of our fixes, but as this post should explain, we are traveling uncharted territory: some of the corner cases we are currently addressing are not sufficiently covered by JLS. Additionally, type inference is inherently sensitive to the slightest of changes. Predicting, which programs will be affected by a given change in the implementation of type inference is near impossible.

Yet, it’s certainly not a game of “them” vs “us”: JLS, javac, and ecj, we’re all in this together, and only by continuing to talk to each other, eventually we will all speak the same language, when we say “Java 8”. Please bear with us as the saga continues …

---

PS: Another pet peeve

I am a type system enthusiast, mostly, because I like how type checkers can completely eliminate entire categories of bugs from your programs. I like to give a guarantee that no code that is accepted by the compiler will ever fail at runtime with an exception like attempting to invoke a method that is not present on the receiver, or class cast exceptions in source code that doesn’t mention any class cast. Type inference is the tool that alleviates the verbosity of explicitly typed programs, while at the same time maintaining the same guarantees about type safety.

Unfortunately, there is a class of Java programs for which such guarantees can not be given: if a program uses raw types, the compiler needs to generate lots of checkcast instructions, to make the code acceptable for the JVM. Each of these instructions can cause the program to blow up at runtime in totally unsuspicious locations.

There are situations where javac silently assumes that a raw type List is a subtype of its parameterized form List<String>. This is wrong. Still I cannot just ignore this problem, because lots of “bugs” are reported against ecj, based on the observation that javac and ecj accept different programs, where in many cases the difference concerns the handling of raw types during type inference.

Economically speaking, investigating in the subtleties of how Java 8 compilers handle raw types is a huge waste of efforts. Any one reading this: if you want to do me a favor, and thus help me to focus on the most relevant aspects of compiler development, please clean up your code. If you keep your code private, nobody will suffer except from yourself, but please, before posting a bug report against JDT, if your code example contains raw types, think thrice before submitting the report. Adding proper type arguments will certainly improve the quality of your code. Likely, after that exercise also ecj will be a lot happier with your code and give you correct answers.

Do I need to repeat that raw types were a workaround for migrating towards Java 5? … that raw types were discouraged starting from day 1 of Java 5? If that doesn’t convince you, search on StackOverflow for questions mentioning raw types and type inference, and you will see that by the use of raw types you are basically disabling much of the power of type inference. Let’s please shed the legacy of raw types.

Eclipse Neon.2 is on Maven Central

It’s done, finally!

Bidding farewell to my pet peeve

In my job at GK Software I have the pleasure of developing technology based on Eclipse. But those colleagues consuming my technology work on software that has no direct connection to Eclipse nor OSGi. Their build technology of choice is Maven (without tycho that is). So whenever their build touches my technology we are facing a “challenge”. It doesn’t make a big difference if they are just invoking a code generator built using Xtext etc or whether some Eclipse technology should actually be included in their application runtime.

Among many troubles, I recall one situation that really opened my eyes: one particular build had been running successfully for some time, until one day it was fubar. One Eclipse artifact could no longer be resolved. Followed long nights of searching why that artifact may have disappeared, but we reassured ourselves, nothing had disappeared. Quite to the contrary somewhere on the wide internet (Maven Central to be precise) a new artifact had appeared. So what? Well, that artifact was the same that we also had on our internal servers. Well, if it’s the same, what’s the buzz? It turned out it had a one-char difference in its version: instead of 1.2.3.v20140815 its version was 1.2.3-v20140815. Yes take a close look, there is a difference. Bottom line, with both almost-identical versions available, Maven couldn’t figure out what to do, maybe each was considered as worse than the other, to the effect that Maven simply failed to use either. Go figure.

More stories like this and I realized that relying on Eclipse artifacts in Maven builds was always at the mercy of some volunteers, who typically don’t have a long-term relationship to Eclipse, who filled in a major gap by uploading individual Eclipse artifacts to Maven Central (thanks to you volunteers, please don’t take it personally: I’m happy that your work is no longer needed). Anybody who has ever studied the differences between Maven and OSGi (wrt dependencies and building that is) will immediately see that there are many possible ways to represent Eclipse artifacts (OSGi bundles) in a Maven pom. The resulting “diversity” was one of my pet peeves in my job.

At this point I decided to be the next volunteer who would screw up other people’s builds who would collaborate with the powers that be at Eclipse.org to produce the official uploads to Maven Central.

As of today, I can report that this dream has become reality, all relevant artifacts of Neon.2 that are produced by the Eclipse Project, are now “officially” available from Maven Central.

Bridging between universes

I should like to report some details of how our artifacts are mapped into the Maven world:

The main tool in this endeavour is the CBI aggregator, a model based tool for transforming p2 repositories in various ways. One of its capabilities is to create a Maven repository (a dual use repo actually, but the p2 side of this is immaterial to this story). That tool does a great job of extracting meta data from the p2 repo in order to create “meaningful” pom files, the key feature being: it copies all dependency information, which is originally authored in MANIFEST.MF, into corresponding declarations in the pom file.

Still a few things had to be settled, either by improving the tool, by fine tuning the input to the tool, or by some steps of post-processing the resulting Maven repo.

• Group IDs
  While OSGi artifacts only have a single qualified Bundle-SymbolicName, Maven requires a two-part name: groupId x artifactId. It was easy to agree on using the full symbolic name for the artifactId, but what should the groups be? We settled on these three groups for the Eclipse Project:
  • org.eclipse.platform
  • org.eclipse.jdt
  • org.eclipse.pde
• Version numbers
  In Maven land, release versions have three segments, in OSGi we maintain a forth segment (qualifier) also for releases. To play by Maven rules, we decided to use three-part versions for our uploads to Maven Central. This emphasizes the strategy to only publish releases, for which the first three parts of the version are required to be unique.
• 3rd party dependencies
  All non-Eclipse artifacts that we depend on should be referenced by their proper coordinates in Maven land. By default, the CBI aggregator assigns all artifacts to the synthetic group p2.osgi.bundle, but if s.o. depends on p2.osgi.bundle:org.junit this doesn’t make much sense. In particular, it must be avoided that projects consuming Eclipse artifacts will get the same 3rd party library under two different names (perhaps in different versions?). We identified 16 such libraries, and their proper coordinates.
• Source artifacts
  Eclipse plug-ins have their source code in corresponding .source plug-ins. Maven has a similar convention, just using a “classifier” instead of appending to the artifact name. In Maven we conform to their convention, so that tools like m2e can correctly pick up the source code from any dependencies.
• Other meta data
  Followed a hunt for project url, scm coordinates, artifact descriptions and related data. Much of this could be retrieved from our MANIFEST.MF files, some information is currently mapped using a static, manually maintained mapping. Other information like licences and organization are fully static during this process. In the end all was approved by the validation on OSSRH servers.

If you want to browse the resulting wealth, you may start at

• https://repo1.maven.org/maven2/org/eclipse/platform/
• https://repo1.maven.org/maven2/org/eclipse/jdt/
• https://repo1.maven.org/maven2/org/eclipse/pde/

Everything with fully qualified artifact names in these groups (and date of 2017-01-07 or newer) should be from the new, “official” upload.

This is just the beginning

The bug on which all this has been booked is Bug 484004: Start publishing Eclipse platform artifacts to Maven central. See the word “Start”?

To follow-up tasks are already on the board:

(1) Migrate all the various scripts, tools, and models to the proper git repo of our releng project. At the end of the day, this process of transformation and upload should become a routine operation to be invoked by our favourite build meisters.

(2) Fix any quirks in the generated pom files. E.g., we already know that the process did not handle fragments in an optimal way. As a result, consuming SWT from the new upload is not straight forward.

Both issues should be handled in or off bug 510072, in the hope, that when we publish Neon.3 the new, “official” Maven coordinates of Eclipse artifacts will be even fit all all real world use. So: please test and report in the bug any problems you might find.

(3) I was careful to say “Eclipse Project”. We don’t yet have the magic wand to apply this to literally all artifacts produced in the Eclipse community. Perhaps s.o. will volunteer to apply the approach to everything from the Simultaneous Release? If we can publish 300+ artifacts, we can also publish 7000+, can’t we? 🙂

happy building!

Runtime Specialization – At Last

Between a rock and a hard place

Not long ago, I had to pull Object Teams out of the Eclipse simultaneous release train. Reason: the long standing issue of using BCEL for bytecode weaving, for which no Java 8 compatible version has yet been released. With the Eclipse platform moving to Java 8, this had escalated to a true blocker. During the last weeks I investigated two options in parallel:

• Upgrade BCEL to a release candidate of the upcoming 6.0 release.
• Finalize the alternative weaver for OT/J, which is based on ASM, and thus already capable of handling Java 8 byte codes

I soon found out that even BCEL 6.0 will not be a full solution, because it still has no real support for creating StackMapTable attributes for newly generated bytecode, which however is strictly mandatory for running on a JVM 8.

For that reason I then focussed on the OTDRE, the Object Teams Dynamic Runtime Environment. This has been announced long ago, and after all, I promised to show a sneak preview of this feature in my presentation at EclipseCon Europe:

Runtime Specialization
Java has never been so dynamic before

Success at last

Today I can report success in two regards:

• The Object Teams Development Tooling, which itself is a complex OT/J application, can (mostly) run on the new runtime!
• I created a first demo example that shows the new capability of runtime weaving in action – it works! 🙂

This is a major milestone! Running OTDT on top of OTDRE is a real stress test for that new component – once again I realize that dog-fooding an entire IDE on its own technology is quite an exciting exercise. While a few exceptions need to be ironed out before the Neon release, I’m confident now, that we’re finally and really on the home stretch of this effort.

But OTDRE is not just a replacement for the traditional runtime, it is also way cooler, as the second success story shows: it is indeed possible now, to throw new teams and roles into a running application. If the application has been generically prepared for this task, the new teams can be automatically activated and immediately start adapting the running application – no specific preplanning needed. With this we are able to achieve a level of dynamism that typically is only possible with dynamic languages. And all this without any compromise to static typing and analysability. This is probably too cool to be allowed.

And after all the hard work on Java 8, also OT/J can finally fully leverage the new version, not only in theory, but also in bytecode.

Less than one week to finalize the presentation. You can be sure this will be a fresh story. Join me on Wednesday, Nov 4,  in Ludwigsburg:

PS: The “traditional” Object Teams Runtime Environment isn’t dead, yet. I really want to keep it as an option, because both variants (OTRE / OTDRE) have quite different characteristics, and after all this component has matured over more than 10 years. But with one option already (mostly) working, I can probably wait until a proper release of BCEL 6.0, and still have it back in game before the Neon release.

Object Teams now has Lambdas (and more)

In a previous post I announced that the Luna version of Object Teams will incorporate support for Java 8.

Now that the big rush towards March 18 is over, I took stock how Object Teams fits into the Luna stream, which finally contains all the bits for Java 8 support. The result:

We have our first builds with full support for OT/J on Java 8!

This means, you can now use lambda expressions and more even in OT/J programs, here’s a witness:

While this program may not make a lot of sense, it demonstrates a funny piece of syntax. Here’s what you see:

• The regular class PersonCollection offers a method restrict that takes an argument of a functional type (Predicate)
• Role Club.Members is played by PersonCollection and makes the base method available via a callout binding (lines 19 f)
• The role wants to pass just an int value, where the base method expects a Predicate.
• This incompatibility is bridged by a parameter mapping that maps a lambda expression to the expected parameter (line 20).

This looks funny, because two different uses of the token -> meet on a single line: the first -> is part of the lambda expression, separating the parameter from the body, the second -> is part of the parameter mapping, mapping the expression on the left to the parameter on the right.

Were I to chose the syntax in a green field situation, I would certainly not use the same token for such different purposes. But since we’re basically merging two independent language extensions I had no choice. After some worries that the combined language might not be parseable I’m much relieved to see it actually working!

A very similar situation actually exists concerning type annotations. More on that in a future post.

Remaining steps towards Luna

Merging the entire BETA_JAVA8 code base from JDT into Object Teams is done, but a few regressions (< 30 out of 84,000+ tests) remain from that exercise. But compared to the original JDT efforts for Java 8 this truly is peanuts 🙂

We only have one hurdle still to jump: our bytecode weaver is based on BCEL, for which no update for Java 8 is in sight. This means: we cannot weave into class files that use the Java 8 bytecode version, bummer!

So the real interesting question will be: can we get the long announced alternate weaver using ASM up to speed in time for Luna? I think we can, but it won’t hurt if you wish me luck in this endeavour.

And yes: one day I should start writing meaningful examples using Java 8 features in OT/J code…

How to kick the fly-shuttle

This post was originally written on August 6th, 2013, but got lost when moving this blog. Luckily I could recover it from backup.

Software composition by “weaving”

Programming languages in the wider field of Aspect Oriented Software Development introduce some kinds of composition concepts, that cannot directly be mapped to pure Java. We got used to speaking of “weaving” as a mechanism for preparing classes in these languages for execution on the JVM, whereas a true aspect oriented virtual machine would natively perform this composition transparently and more efficiently. But with the dominance of the JVM it will be “weaving” for a long time to come.

Back in the early days, AOSD languages performed some static composition on source or class files. When OT/J entered the stage, first frameworks emerged that allowed us to hook into the class loading process, so we could do the weaving dynamically on-the-fly, without ever storing the woven classes on disk. Work is (still) in the pipe-line to support even runtime-(re)weaving.

One theme in all the development of Object Teams is: first solve the dynamic case, that’s where the challenges lie, and leave more static cases for later as an optimization. “Later” has come now.

How (not) to hook into class loading?

A little while into the development of OT/J, Java 5 was published introducing the JPLIS API and the concept of a -javaagent. Today many instrumentation tools use this facility and OT/J was among the first on this boat. So for standalone Java applications the problem is solved.

Later, while developing Eclipse plug-ins, we noticed that it wouldn’t be overly cool if you have to invoke Eclipse with a -javaagent. Not only is this very inconvenient for installation, it is also problematic (e.g., for performance) to feed all classes of Eclipse through our weaver. We were lucky again, and the Equinox Adaptor Hooks emerged right when we needed them. With these hooks we were able to integrate OT/J in a way that enables all the on-the-fly flexibility we want while at the same time it lets us control the process so that the flexibility cannot be abused. So in Equinox (or better: OT/Equinox) the problem is (more than) solved.

The fact that Eclipse Luna will no longer support these hooks was slightly alarming news for Object Teams, but that problem is already essentially solved – details to be reported soon.

I’ve heard about successful experiments running OT/J code in tomcat, but, hey, the list of application servers/containers is long, and if we need specifically crafted integration for each environment, it’s kind of difficult to argue that OT/J runs “everywhere”. Is there a generic why to kick the fly-shuttle?

Dynamic weaving is cool, but not having a fallback is uncool, so I finally developed the static scenario as a special case of the dynamic one: a build-time weaver for OT/J.

The little new tool is ridiculously simple: we have a weaver capable of load-time weaving. All I had to do is: create a little utility that would invoke this weaver ahead-of-time, i.e., during building.

While I’m not a big fan of Maven, in my day job Maven is a must, so I bit the bullet used the opportunity to learn a bit about development of Maven plug-ins. I was amazed to find lots of tutorials still suggesting to use API that are deprecated in Maven 3 and more surprised to read forum threads suggesting to still use that deprecated API, partly to be compatible with Maven 2 (OK?) and partly because deprecation doesn’t always mean there is an alternative (OOPS?). It’s also interesting to see, that much fundamental, up-to-date information is not available from a central source but only from tribal knowledge in hundreds of forums, blogs and whatnots. — Enough whining, on to the new stuff.

For simple projects, all you need now is a pom that looks about like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 
    <modelVersion>4.0.0</modelVersion>
 
    <parent>
        <groupId>org.eclipse.objectteams</groupId>
        <artifactId>objectteams-parent-pom</artifactId>
        <version>2.2.0</version>
    </parent>
 
    <artifactId>OTStopwatch_Built-Time_Weaver_Example</artifactId>
    <version>0.0.1-SNAPSHOT</version>
 
    <properties>
        <tycho.version>0.18.0</tycho.version>
        <otj.version>2.2.0</otj.version>
    </properties>
 
    <build>
        <plugins>
            <plugin>
                <groupId>org.eclipse.objectteams</groupId>
                <artifactId>objectteams-weaver-maven-plugin</artifactId>
                <version>0.8.0-SNAPSHOT</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>weave</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <teamClasses>
                        <teamClass>org.eclipse.objectteams.example.stopwatch.WatchUI</teamClass>
                        <teamClass>org.eclipse.objectteams.example.stopwatch.WatchUIAnalog</teamClass>
                    </teamClasses>
                </configuration>
            </plugin>
        </plugins>
    </build>
 
    <repositories>
        <repository>
            <id>ObjectTeamsRepository</id>
            <name>Object Teams Repository</name>
            <url>http://download.eclipse.org/objectteams/maven/3/repository</url>
        </repository>
    </repositories>
 
   <dependencies>
       <dependency>
           <groupId>org.eclipse.objectteams</groupId>
           <artifactId>objectteams-runtime</artifactId>
           <version>${otj.version}</version>
       </dependency>
   </dependencies>
</project>
 

Here we enable OT/J compilation via the parent pom, specify a repo where to get the OT artifacts, a dependency and a few versions, nothing sophisticated.

What’s new is the objectteams-weaver-maven-plugin, which is minimally configured just by specifying a list of team classes that should be woven. Oops, that’s already it.

By integrating this into your build, you’ll produce class files that can be directly executed “everywhere”, i.e., without the OT/J load-time weaver, and thus: without fiddling with the class loading process. One caveat remains: naturally, by producing the woven class files offline, it will be your responsibility to correctly feed the correct versions of class files into the classpath in the desired order. If you can’t use the load-time weaver, it won’t be able to help keep things simple.

More configuration options are documented on the plug-in’s site and more explanation using our favorite examples can be found in the wiki.

Here you have it: static weaving falls of naturally if you already have dynamic weaving. And the resulting class files can truly be run “everywhere”.

I’d be curious to hear of the first OT/J apps on Android (though personally I’d prefer ubuntu).