So in the end I did get around to finding a way to extract those mystery JDK6 "bin" files. It turns out they're just heavily disguised zip files. Very heavily disguised - looking at the content of the files first points to being shell scripts, then if you look closer you'll see that they're in fact embedded ELF executables. The only way to tell as far as I could figure out to discover that they're really zips was to actually try unzipping them and discover to your astonishment that it works.
So based on Andrew's code I cobbled together a script using a combination of wget, perl and shell that'd go identify the most recent jdk6 (or theoretically jdk7) download; extract the relevant bits, and produce a japi file from it.
It's still a bit rough round the edges but preliminary results are available: Classpath Generics versus JDK6
and Classpath Generics versus JDK7
. Well actually they're not yet available as I write this, but the script that'll hopefully produce them is running as I type. So perhaps they'll be there by the time you're reading this. If not, maybe I screwed up the scripting... but it's the thought that counts, right?
These japi files are also produced with the aid of a new trivial program japiextractpkgs, which parses the overview-frame.html file from any set of Javadocs to produce a list of packages which, thanks to Jaroslav's contribution, Japize can now understand. I haven't yet switched the Japi file creation for existing JDK versions to use this tool, but I will be soon once I'm completely confident it's doing the right thing.
One more step towards Japi perfection. Another step towards Japi perfection comes from David Fu contributing an HTMLWriter implementation to Classpath - this being the last completely missing class from 1.2, 1.3 and 1.4. That's a pretty big deal...Update
a few hours later: Looks like JDK6 worked but JDK7 didn't. Will fix tomorrow. Also, forgot to mention that the inspiration for the right approach to investigate the file format in the first place came from James Stansell's comments on the last post. Thanks James!