I just discovered a nasty way in which the new for syntax fails with older code. Since the new for loop depends upon its target expression to implement the Iterable interface, older code which implements the pre 1.5 Collection interface didn't have to supply an Iterator that extended the SimpleIterator interface. So, when you use the new for loop on an older Collection, you end up getting a runtime error that the abstract method

SimpleIterator Collection.iterator()

hasn't been implemented.

This bug is pretty heinous, because there is no way to detect it at compile time. You run into the same sort of problems when you deal with JDBC drivers, for example, but that's not anywhere as near as big a problem as this is. Is their any plan to deal with this behavior? (Neal, anyone?) Currently, I have a little adapter class:

class ForAdapter<T> implements Iterable<T> {
  public ForAdapter( Collection c ) {
    // ...
  }
  // ...
}

that I use like so:

for ( String s : ForAdapter<String>( foo() ) ) {
  // ..
}

but that's kind of nasty. If there are no plans to deal with this behavior, then I'll probably write a bytecode enhancer that I can run on 3rd party libraries that will patch them to work with the new for loops. At least, then, I can guarantee that I won't get any unexpected runtime exceptions. Anybody have any better ideas?

God bless,
-Toby Reyelts

 

 

In case anybody's curious, I've actually written up the utility. It's pretty simple. You just use BCEL to add the following method to any Collection implementing class:

public java.lang.SimpleIterator iterator() {
  return new Iterator15Adapter( iterator() );
}

where the call to iterator() is on the old iterator method. Iterator15Adapter is just a class that takes an Iterator instance and implements the SimpleIterator interface with it. This works, because the VM spec allows this kind of overloading, even if the Java language doesn't.

God bless,
-Toby Reyelts

 

 

This sounds like a compiler bug. The iterator() in Collection overrides the one in Iterable, so anything that already implements Collection should work with the new loop.

 

 

This sounds like a compiler bug.

Isn't this, rather, a problem with the compiler-based implementation of covariant return types? Since Iterator<E> Collection.iterator() is covariant to SimpleIterator<E> Iterable.iterator(), it means that, under the 1.5 compiler, a bridge method for SimpleIterator<E> iterator() would be generated in any class implementing Collection<E>. (From what I can tell, all of the 1.5 Collection classes do indeed have this bridge method. The lack of a working decompiler makes this difficult at best). Classes compiled pre-1.5 wouldn't have that bridge method. When no bridge method exists, the virtual machine keeps walking the inheritance hierarchy to find a match (which means that the overrider's version of the method will be totally skipped - Yuk!). In the particular case I'm discussing here, the class implementing the Collection interface has no base class directly implementing SimpleIterator<E> iterator(), and so the VM can't find any match which results in an AbstractMethodError.

Basically, covariant return types don't work with code compiled across different (pre-1.5, 1.5+) compilers. Seems like a pretty serious problem to me. If we changed the invokevirtual instruction so that it were actually smart enough to understand covariant return types (instead of relying on broken compiler-trickery), we wouldn't have this problem. I understand Sun's reluctance to change the virtual machine, but you're going to be creating a new one for 1.5 anyway, so why not just go ahead and fix the behavior?

As far as my little bytecode-enhancing utility goes, I guess this means I can pare the generation down to a bridge method and exclude the useless adapter class.

God bless,
-Toby Reyelts

 

 

The lack of a working decompiler makes this difficult at best.

That should read: The lack of a working disassembler makes this difficult at best. I'd be quite happy with a javap that works on 1.5 compiled code.

God bless,
-Toby Reyelts

 

 

Doesn't SinjP?

 

 

SinjP doesn't give you the bytecode for the class file - just the signature information.

I learned from Scott earlier today that the FLEX framework that SinjP uses does come with a tool that can dump assembly, though. It's not very pretty, but it's better than nothing.

God bless,
-Toby Reyelts

 

 

Ouch. Yes, it is. This is a very serious problem. We may have to abandon our use of covariant returns here.

To make a long story short, because the JVM represents a method using a signature that includes the return type, this retrofit had the effect (from the VM point of view) of adding a method to Collection. This method is normally created silently by the compiler so the user doesn't have to worry about it - it's called a "bridge" method. But implementations of Collection that were compiled using a previous JDK lack this bridge method, causing them to fail (AbstractMethodError, I believe) at runtime. We've violated the basic rule that one cannot add anything to an interface once it's been made public.


It would be incompatible to change the semantics of existing classfiles. And that's what you have: an existing classfile.

 

 

We may have to abandon our use of covariant returns here.

Urr.. what exactly does that mean? Your use of the term abandon doesn't exactly give me warm fuzzies. By abandon, I assume you mean removing SimpleIterator from Iterator and Iterable from Collection. I'm not sure how you would continue to support the new for loop (very special compiler support?), but I don't like the idea that an elegant solution is forsaken and that the underlying problem isn't solved.

One option is to build a trampoline into the compiler-generated for loop to guarantee that you get a SimpleIterator from a Collection. Of course, this only treats the symptom (and only one symptom - albeit the most likely by far) and not the problem.

Another option is to change the VM so that, at class load time, any methods with covariant return types automatically gain bridge methods if they don't exist. This would have to be a VM spec change, but it's probably more innocuous than a change to invokevirtual.

I'm sure there are other options.

It would be incompatible to change the semantics of existing classfiles. And that's what you have: an existing classfile.

The Java Language Specification is very strict about specifying that you can't declare two methods with the same signature but different return types. Given that, I don't see how depending upon invokevirtual to not find matches of methods with the same signature is valid for bytecode that is derived from a Java program. (Yes, I see the irony in the fact that the same feature I'm saying has no use in the Java language is the very feature currently used to implement covariant return types).

Granted, someone may have been doing something goofy with some other language targeting the JVM, but

a) Sun doesn't care about other languages targeting the VM.

b) I'm sure those who target the VM would be quite willing to pay the price of a mostly benign change to get true support for covariant return types in the upgrade to 1.5.

God bless,
-Toby Reyelts

 

 

All I meant is get rid of SimpleIterator and use java.util.Iterator instead. Iterable would remain. But now I'm thinking this was just a code gen bug in the compiler. The compiler generated a call to Iterable.iterator() when it should have generated a call to Collection.iterator(). This is in line with the requirements of the binary compatibility chapter.


This doesn't help - an Iterator IS a SimpleIterator. The problem is that the compiler is generating a call to the wrong iterator() method.


The JVM spec doesn't require that the class file be derived from a Java program.


False.


I'm not so sure that the change is benign or that it would be widely acceptable.

 

 

But now I'm thinking this was just a code gen bug in the compiler.

I don't think so, even though I'm all for a quick fix. You would have to explicitly check that the expression resolved to a Collection (as opposed to just an Iterable) so that you could call Collection.iterator(). What happens when I have a method that returns an Iterable, that really returns a Collection class, that was compiled pre-1.5:

public Iterable foo() {
  return new Pre15ClassImplementingCollection();
}
public void baz() {
  SimpleIterator it = foo().iterator();
}

Won't that still result in an abstract method error? Yuk.

This doesn't help - an Iterator IS a SimpleIterator.

I think you missed my point. You would wrap the iterator returned from a call to iterator() with one that you know was compiled with 1.5 and therefore has a bridge method. It was an ugly solution (as I stated), but effective in making sure the for loop doesn't bomb.

The JVM spec doesn't require that the class file be derived from a Java program.

I never said it did. I did say that the only supported target for the JVM is the Java language.

False.

This must be a case of one hand not knowing what the other does, because every statement I've ever seen made has been to the contrary. Not that I don't personally want Sun to support other languages targeting the JVM - they just never have.

I'm not so sure that the change is benign.

I believe changing invokevirtual is the right way to solve this problem. Given that, it's your job to figure out whether the right solution is the best one.

or that it would be widely acceptable

It's somebody else's job at Sun to figure this out, but I can at least vouch for myself and a signifcant amount of others. ;)

 

 

Rather, we should always use the qualifying type in the bytecode, as required by the JLS.


Yes it would.


You may have said that, but its not true. The JVM supports the spec described in the VM spec, not matter what language it originally came from. If the JVM doesn't obey the spec, then its a bug and we'll fix it.


Right, we don't support the languages, but we DO support the VM, and the fact that it obeys its spec no matter how you generated the class file.


Your belief is not sufficient proof that the change is backward compatible.

 

 

Yes it would.

And your resolution?

You may have said that, but its not true.

Neal, this is becoming silly fast. I'm saying that Sun only supports the Java language. I'll be glad to examine any proof to the contrary.

Your belief is not sufficient proof that the change is backward compatible.

Well, unlike you and your fellow employees, I don't get paid to determine if it is backwards compatible. If however, you decide that you'd like to pay somebody who is well-versed in Java bytecode, the Java language, compilers, and computer architecture in general, to perform a study to determine the most appropriate solution for solving this covariance problem, I can be consulted for a modest fee. Otherwise, I have more pressing things to attend to.

God bless,
-Toby Reyelts

 

 

One solution is: If it hurts, then don't do that. It is yet to be seen how we will address this issue, but another option is to do away with the class SimpleIterator and use java.util.Iterator instead.


Sure. Sun also supports the JVM, which has a separate specification and can be (indeed often is) targeted by other languages.


If you believe it is my job to provide proofs for your theories, then you misunderstand my job.

 

 

If it hurts, then don't do that.

That might make sense if it were easy and intuitive to avoid "doing that". That is apparently not the case, since the designers of the language compiler and libraries can't seem to get it right. This is because there is a pretense of binary compatibility, that doesn't hold with respect to one feature of the language.

One solution is...

To repeat myself, that is a treatment for a symptom, and not a resolution to the problem.

Sun also supports the JVM

I'm quite aware of other languages which have been targeted to the JVM, which is why I'm also quite aware that Sun makes no effort to suit the JVM for use with other languages. I'm not suggesting that you support the JVM with any less fervor. To the contrary, I'm suggesting that you continue to support it as you currently do - even moreso.

If you believe

If you believe that it is not your job to solve the problem in the best manner in which you are capable, perhaps you should be looking for another job? To forego even preliminary investigation of a simple and elegant remedy to a problem seems negligent, but that is your decision to make.

God bless,
-Toby Reyelts