... indistinguishable from magic
effing the ineffable since 1977


Recent Posts


Japi Feet

Well, the Arrays bug is fixed and should be in tonight's Japi run, unless I messed up another fix for the "-p" flag. But I can't figure out what's up with DefaultTextUI. When I Japize it with the JDK on Windows it works fine, but with gcj on Ubuntu I get this with the latest cvs japitools:

sballard@tinian:~/japitest$ wget
sballard@tinian:~/japitest$ japize unzip as dtui packages =javax.swing.text

Processing package javax.swing.text,:=+++--+-++++++++-++++++ ++++--+++----+-+---+++-------+++-+-----------------+-+++--+++-++-
Failed to Japize javax.swing.text.DefaultTextUI: java.lang.NoClassDefFoundError: javax.swing.text.DefaultTextUI
java.lang.NoClassDefFoundError: javax.swing.text.DefaultTextUI
   at net.wuffies.japi.ClassFile.forName(
   at net.wuffies.japi.Japize.getClassWrapper(
   at net.wuffies.japi.Japize.japizeClass(
   at net.wuffies.japi.Japize.processPackage(
   at net.wuffies.japi.Japize.processRootSet(
   at net.wuffies.japi.Japize.doJapize(
   at net.wuffies.japi.Japize.main(
++++++--+-++-+-----+-++++++++--+--+++++++++++-+++ +++++-----++++++++-+-++++-+++++-+++++++++--++++--+-
sballard@tinian:~/japitest$ java -fullversion
java full version "gcj-1.4.2"

The fact that this works on the JDK suggests it's not a ClassFile bug as I originally thought. But I'm not sure what kind of bug it is. Does gcj have known issues in And why would they trigger on this file only?


If it makes you Japi...

I've just made a few more tweaks to the nightly Japi scripts. This may result in some false positives in the nightly diff emails, or it may result in the nightly diff emails being completely screwed up or missing entirely due to me having messed up the changes. Here's what I've done:
  • On an entirely trivial note, cleaned up the nightly processing script to move a bunch of repetitive code into functions. This shouldn't have any concrete effects but made the other changes, and hopefully future changes, easier to implement.
  • Added reverse comparisons against jdk6 and jdk7 in addition to the forward ones that were already happening. (Those links will be broken until such time as the run happens successfully)
  • Changed the Japize process for all the JDK versions from 1.2 up to use the new japiextractpkgs tool to pull the package list from their documentation. I'm pretty sure I'd got these right by hand, but automating the process doesn't hurt and certainly saved time getting jdk6 and 7 right.
  • Added a new "-p" flag to japicompat. This is intended for use in the reverse comparisons (classpath-vs-jdk15 for example). Up to now, if the classpath japi file had included any packages that aren't part of jdk15, those would have been reported as "missing in jdk15" and skewed the percentages, so it was important to make the list of packages in the classpath japi file match jdk15 exactly and exclude any extra packages. On the other hand this made it impossible to get a true result against jdk6 even in the forward direction, in case some of those packages that got excluded were actually needed for jdk6. With the "-p" option, japicompat will simply ignore any missing packages and calculate the percentages accordingly.
While working on these changes I've discovered a couple of bugs: JDK6's java.util.Arrays class cannot be Japized - apparently it uses an array type in some context where Japize believed arrays were illegal - and neither can classpath-generics's javax.swing.text.DefaultTextUI (Japize simply complains about "class not found" - but if it didn't find it, how did it even know to look?)

I'm sick this weekend which means (a) lots of relaxation trying to persuade my body to recover fast, but (b) possibly a bunch of free time to hack on this stuff. I may even get to working on Annotation support which is the last 1.5 language feature missing in japitools and the big showstopper for a 1.0 release.


Many Japi Returns

So in the end I did get around to finding a way to extract those mystery JDK6 "bin" files. It turns out they're just heavily disguised zip files. Very heavily disguised - looking at the content of the files first points to being shell scripts, then if you look closer you'll see that they're in fact embedded ELF executables. The only way to tell as far as I could figure out to discover that they're really zips was to actually try unzipping them and discover to your astonishment that it works.

So based on Andrew's code I cobbled together a script using a combination of wget, perl and shell that'd go identify the most recent jdk6 (or theoretically jdk7) download; extract the relevant bits, and produce a japi file from it.

It's still a bit rough round the edges but preliminary results are available: Classpath Generics versus JDK6 and Classpath Generics versus JDK7. Well actually they're not yet available as I write this, but the script that'll hopefully produce them is running as I type. So perhaps they'll be there by the time you're reading this. If not, maybe I screwed up the scripting... but it's the thought that counts, right?

These japi files are also produced with the aid of a new trivial program japiextractpkgs, which parses the overview-frame.html file from any set of Javadocs to produce a list of packages which, thanks to Jaroslav's contribution, Japize can now understand. I haven't yet switched the Japi file creation for existing JDK versions to use this tool, but I will be soon once I'm completely confident it's doing the right thing.

One more step towards Japi perfection. Another step towards Japi perfection comes from David Fu contributing an HTMLWriter implementation to Classpath - this being the last completely missing class from 1.2, 1.3 and 1.4. That's a pretty big deal...

Update a few hours later: Looks like JDK6 worked but JDK7 didn't. Will fix tomorrow. Also, forgot to mention that the inspiration for the right approach to investigate the file format in the first place came from James Stansell's comments on the last post. Thanks James!



When a lot of things are happening that would be good to blog about, there's never enough time to blog about them. And when nothing particular's happening and there's time to catch up on the stuff you didn't blog about earlier, it all seems like old news, so why bother?

I've been in my new apartment now for a few weeks and am settling in nicely. It's strange to have nobody to answer to but myself - I'm still getting used to that. It's even stranger to have entire days when the only criterion determining what I do is what I decide needs to be done. This entire weekend, in fact, has been like that: there were things that needed to be done, but not because anyone else said so.

On September 30th I went to see a midnight showing of Serenity. I'd hoped to win a Jayne hat in the trivia, but unfortunately (a) the Jayne hat turned out to be the prize for best costume, and I wasn't in costume, and (b) the only trivia question I got my hand up in time to answer, I answered completely wrong. My hand was working faster than my brain, clearly. I knew the right answer, just not quickly enough. I did have the foresight to ask whether it was okay to take a plastic dinosaur - but not the foresight to take two so that I could actually enact the scene... guess I need to buy a second one for myself. Still it was a lot of fun, as expected. And hopefully contributing to making Serenity profitable so that a sequel, or better yet a new series, gets made... please?

In the meantime I'm working my way through Buffy, which appealed to me in absolutely no way whatsoever until I discovered Firefly by the same guy (Joss Whedon, for anyone living under a rock - who also wrote the script for Toy Story). After falling head over heels in love with Firefly I had to forget about my preconceptions and give Buffy a chance, and of course it turns out to be as good as you'd expect from a Joss show. Season 1 is good, but things are really picking up in Season 2, and I'm told it only gets better from there.

I wonder if my posts describing Japi progress on a regular basis were actually helping the numbers to go down as fast as they did. Work on Classpath is proceeding as enthusiastically as ever, but since around the same time I stopped giving regular Japi updates, it seems more focused on bug-fixing and polish and other "make things work right" aspects than on pure API completeness. Japiwise, progress is being made, but not as fast as it was.

The status right now is that Classpath is API-complete for 1.1 again (it was, then the ability to mark methods as not-implemented came along; they're all now implemented). For 1.2 and 1.3 the status is almost identical: one class (javax.swing.text.html.HTMLWriter - work in progress, I believe) and 62/63 methods missing. (The discrepancy, for what it's worth, is the eloquently named javax.swing.text.html.HTMLEditorKit.InsertHTMLTextAction.insertAtBoundary method - just be glad I didn't include all the parameters too). The missing methods are mainly concentrated in java.awt.font, java.beans.beancontext and javax.swing.text.html. There are also a few in java.awt.datatransfer, java.awt.dnd and java.text which might be low-hanging fruit. 1.4 completeness isn't that much further away - only another 25 or so methods scattered across various packages. Amazingly, while nobody was looking the generics branch has snuck past 95% completeness against 1.5. Not so long ago it was below 90% - that's incredible progress!

With the divorce and the moving and my own general lack of organization I've been a very lax maintainer of Japitools - and my other open source projects - lately. When I was on vacation in England back in June (!) I received two very useful contributions. Firstly, from Jaroslav Tuloch of Sun I received an Ant task for Japize, some JUnit tests, and a patch for allowing Japize to process packages non-recursively - that is, include a package but not its subpackages. This last one could be enhanced to provide something I've wanted for ages - the ability to extract lists of packages from Javadoc documentation, so that my nightly scripts would no longer need to hardcode the list of packages for each JDK version, but could pull them from Sun's docs. I finally got around tonight to actually taking this code, compiling it, checking it into cvs and doing minimal testing on it. Currently one of the ant task tests fails, but I'm not sure why. In an ideal world I'd test it more, but holding out for perfection will just lead to another 4 month delay. So it's in now, and if I broke anything, we'll see the results in the nightly runs.

The other contribution was from Andrew John Hughes - a script to go to Sun's website and identify the filename of the latest JDK1.6 beta release, and download it. I'm just looking at that for the first time tonight too, and I have a dilemma. Sun helpfully provide the JDK only in self-extracting ".bin" format. I'm sorry, but I have no intention of downloading and executing arbitrary code unsupervised from a nightly script. Especially since it's not an https server, so there's no protection against DNS spoofing. Does anyone know of a tool which can extract this "bin" format without running it? They do provide the JRE in a jar file, but the JRE generally doesn't include any indication which methods are deprecated, so the Japi results would give erroneous errors about deprecation.

Anyway I'm hoping that by shaming myself publically by admitting how terribly slow I was to respond to these great contributions, I'll be more motivated to not do that in future. Maybe.