... indistinguishable from magic
effing the ineffable since 1977


Show All

Recent Posts


Things that just should exist


The IKVM FAQ says that one of the main reasons for its existance is that it "just seems obvious that something like this should exist". That phrase struck a chord with me because I completely agree - if it didn't exist, I have the impression that the universe would have a distinctly IKVM-shaped hole in it.

In the last couple of weeks I've been working on a couple of possible small features (coincidentally, for IKVM) which leave me with the same impression.

The first one I only implemented a tiny fraction of - which happens to be the exact tiny fraction I need.

Some background: nrdo (which I run through IKVM) connects to your database at compiletime and creates your tables for you. However, since it's a Java program, it does that through JDBC, which isn't ideal if you're working on a .NET project for several reasons. Firstly, you need an IKVM-compiled version of the JDBC driver for your database. If your database (and driver) are proprietary, this constitutes a modified version of proprietary code and is therefore probably illegal to redistribute. Secondly, you need to specify your database connection information twice, once as a JDBC URL for use at compiletime, and once as an ADO.NET ConnectionString for use at runtime. And thirdly (at least if SQL Server is your database), the JDBC driver requires you to enable TCP/IP connections, while the ADO.NET provider can use some magic Windows protocol which can be better in various ways from a security standpoint (unlikely as that sounds).

So what I implemented was an implementation of the (very few... okay three) JDBC functions that nrdo actually calls, implementing them using ADO.NET underneath. Just stick "jdbc:sharp:System.Data.SqlClient:" in front of your ADO ConnectionString and you have a JDBC URL for the same database, as long as you don't want to do anything overly complicated like, I dunno, getting any results back from your queries.

This definitely seems like "something that just obviously should exist", but since it met my immediate need I don't have any time or motivation to add the other 99% of the JDBC driver API. I offered it to Jeroen for inclusion (and possible further development) as part of IKVM proper, but he (reasonably enough) doesn't want perpetually unfinished code rotting in his CVS repository. From his email (quoted with permission):

I have had the opposite (an ADO.NET wrapper for JDBC drivers, written by Mainsoft) sitting in my mailbox for a long time. It would definitely be nice to include both of these, but realistically, I don't have time to maintain them (which is why I never got around to doing anything with the ADO.NET wrapper) and I don't want to put them in cvs just to sit there and rot away.
So, yes I'd like it, but you'd also need to volunteer to maintain it ;-)

So if anyone has the need, interest or motivation to maintain and finish that code, there's a JdbcSharp-shaped hole in the universe waiting for you ;)

The second hole in the universe I'm trying to fill is something I noticed the very first time I ever used IKVM. I immediately wrote a comment on the IKVM blog about how schizophrenic it seemed to be writing code in C# that looked like this:

for (Iterator i = myList.iterator(); i.hasNext(); ) {
object o =;

It's marginally weird working with Java APIs through IKVM in general because they adhere to a different naming convention than other .NET APIs, but you can at least pretend the author just didn't read the Framework Design Guidelines. And if you're working in an IDE you don't even need to change the way you type - the casing will be corrected for you. But the moment you need to deal with any kind of Java collection you slam into a mental speedbump - you can't get at the data in the list in any way that any .NET programmer would ever think of. You have to switch gears and - for all practical purposes - write two lines of Java code in the middle of your C# program.

Congratulations, you just hit a foreach-shaped hole in the universe.

So I've been working with Jeroen to figure out what would be necessary to make C# foreach (and VB "For Each") work on Java collections when exposed through IKVM. We've got the code working to the extent that this code works:

java.util.ArrayList list = new java.util.ArrayList();
foreach (object o in list) {

But this code doesn't:

java.util.List list = new java.util.ArrayList();
foreach (object o in list) {

because the extra mapping is only done on classes, not interfaces. Stay tuned: I'm also planning on tackling the other direction (which is harder), so that if you have a 1.5-compatible Java compiler you'll be able to do:

cli.System.ArrayList list = new cli.System.ArrayList();
for (Object o : list) {


The Unix philosophy of Lego bricks


As a child I spent an inordinate amount of time playing with Lego (was it just me or did anyone else invent names for the bricks? I had Long-ers, Square-ers, Tippees, Flat Tippees, Very Flat Tippees, Tipping Up Tippees...) The fact that a tubful of basic components could become the most fabulous spaceship I was capable of imagining had a huge influence on me and was probably a large factor in why I became a programmer.

I really think I'm onto something with cmeScribble but I've had a hard time coming up with a way to explain exactly what I think I'm onto. Part of that is because, obviously, I'm writing largely for an audience that isn't familiar with cmScribe, the philosophy it tries to embody, the strengths of its approach or the limitations we're starting to hit. Perhaps I should let my ten-year-old self explain by analogy.

We needed a powerful layout system so we came up with Lego.
Then we needed flexible hierarchical navigation so we built in Duplo.
Then we needed some place to put the content so we made a Rubik's Magic.
We built an eCommerce system out of Meccano.
We built a document repository out of Rubik's Snakes (those were great, weren't they?)
We made a general-purpose form builder out of MegaBloks.
We made a permissions system out of whatever that cool thing was that had all the translucent octagonal pieces with slots in the sides.

And every time we add a new feature we get a different type of construction toy to make it out of.

We've done a good job of making all these things work together. To display products in the nav we built an adapter that allowed Meccano to attach to Duplo. We built adapters so that octagons could be attached to anything. We built adapters to connect Meccano to Rubik's Magic. We built adapters and adapters and more adapters.

What we've ended up with is something that's extremely flexible and powerful, but in doing so we've solved the same problem lots of times - making one kind of toy talk to another, or making one toy appear as if it's a different kind of toy, or solving a problem for one kind of toy that was already solved for one of the others (notably, our Rubiks-Magic search is entirely separate from our Meccano search).

The idea behind cmeScribble is "what would happen if we just did it all with Lego[1]?"

The Lego philosophy and the Unix philosophy have a lot in common. In cmScribe we got the bit about "Do one thing and do it well" right, whether the one thing is "match a regular expression" or "be two by four, full height and red" or "display a product list". But just as important, if less expressable in a pithy slogan, is that the reason Lego and Unix work as well as they do is that every part has a common interface. In Unix it's the plain text file with each line representing some kind of value. In Lego it's the little round bump and the little bump-sized hole. In both cases the power comes from the fact that any piece's bumps can attach to any other piece's holes without having to create a custom Long-er-To-Very-Flat-Tippee-Adapter.

In the next thrilling installment: What do I propose as cmeScribble's little-round-bump-equivalent?

[1] Lego is, after all, the One True Construction Toy and MegaBloks can bite my small cuboidal plastic butt ;)


Are you mocking me?


Before I wrote a line of code for cmeScribble I decided that I'd code it the right way. Retrofitting "right-way-ness" onto an existing project is almost impossible, so I was determined not to miss the chance to build it in from day one.

The particular "right way" I had in mind was using Test-Driven Development, or TDD. For anyone unfamiliar with the TDD methodology, the idea is that before you write any code or fix any bug you first write a test or two to make sure the code works or the bug is fixed. The test should fail when you first run it, because the code doesn't work and the bug hasn't been fixed. Once you've written the test and verified that it fails, then you write the code to make it pass.

The advantages of TDD are well-documented elsewhere, but in a nutshell: writing the test forces you to know in advance exactly what you want the code to do, and also as you develop more and more code you have a library of tests to make sure you don't break anything.

I could easily see how to write tests for self-contained functions, but I couldn't figure out how to write tests for user interfaces or database-backed logic. Since cmeScribble is all about building user interfaces from database data, this posed something of a challenge. Maybe I can get away without testing the UI layer, but I can count features that won't need database data in some way on the fingers of Captain Hook's right hand. Okay, I thought, Google to the rescue... but searching for databases and TDD provided lots of people with similar issues and no straightforward answers.

A common suggestion was to use "mock objects". The idea is that you create a "fake" version of your database layer which returns plausible values without actually needing a database. This solves the problem but the downside is that you have to actually write the mock version of each database table, and since cmScribe has over a hundred tables at last count[1] I didn't relish that thought much either.

Facing a potential deal-breaker for something you don't want to compromise on tends to focus the mind, and eventually I hit on a solution. A full SQL database is hideously complex, but cmeScribble will be doing it's database access through nrdo (I'm biased, but I honestly don't see how people can work any other way...) and for 99% of uses, nrdo provides a fairly simple object model over the top of the database. Simple enough that if you don't care about persistence (which in a test situation is actually a bad thing) or performance, it's possible to implement the vast majority of it very simply in memory, as part of the nrdo runtime library and generated code, backed by simple List<T>s. A couple of days of hacking later and that's exactly what I've now checked into nrdo's CVS: If you use the C# 2.0 template and set the configuration setting "NrdoMock" to "True", all your code will run purely in memory, with a clean "database" for each run.

There are a few things missing - at a wild guess I'd say it covers 60% of what you'd want to do right now, and I should be able to get that up to about 90% without too much trouble by filling in things as I need them (after writing tests!). Most of the rest can't be handled automatically (because they'd require a full SQL engine in the mocking layer) but can be done with a little manual intervention (by the programmer providing a C# equivalent of the SQL clause in question; the SQL clause itself doesn't get tested but all the logic around it does).

The most satisfying part of the whole thing was watching it pass the tests I'd prepared in advance :)

[1] cmeScribble will have a lot less because of the "enterprise" features that aren't being included and also hopefully by unifying lots of things that were separate tables in cmScribe, but still.


cmeScribble - coming soon!


Last night I came up with the seed of a way to implement some things in cmScribe (the CMS that I work on as a day job and happens to run my website too) that I've wanted to do for ages but not known how - and get a bunch more cool features that I never even thought of before for free. That was the good news. The bad news was that the ideas are a radical departure from the way cmScribe works now and would require an almost complete rewrite - only a few pieces of code could be salvaged anything like intact. It's taken us three years of team development to get cmScribe to where it is now - which is, in my not-entirely-unbiased opinion, a pretty good place in lots of ways - so the idea of throwing all that away and starting over based on a mad programmer's wild and unproven blue-sky ideas was probably not something I could sell to the boss ;)

(Basically, the entire architecture of cmScribe today revolves around building templates and pages out of building blocks that we call panels. The idea I had leaves no concept of a panel left at all, and templates and pages don't have any independent existence but just become particular ways of combining lower-level ideas...)

But I think I found a way to turn the bad news to my - and NetReach's, and maybe your - advantage. Because it does mean a complete rewrite, there's no need to use any existing cmScribe code to implement it. So I suggested to my boss that I'd be happy to work on this in my own time as a hobby project / technology testbed, and wouldn't it be cool to do it as an Open Source project? As a hobby project it wouldn't even attempt to provide the "enterprise" features that cmScribe does, but if it successfully proves the concept, it would also be possible to use it to rebuild those enterprise features and get a next-generation cmScribe. My boss agreed.

So without further ado, I give you...

Well, actually, I don't, because all cmeScribble is right now is a logo, a pronunciation (see-me-scribble!) and some partly-formed ideas. There are parts of the design that are defined as being done "by magic". And while some incredibly rich and powerful scenarios are enabled by this architecture, I'm a little fuzzy on how to implement obscure things like, say, pages.

When I have some code (I've started, but I don't want to release until it can do something that at least gives a hint of the power to come) I'll put it online for people to try out. In the meantime I'll blog a bit about the process by which I arrived at the new design, where I think I'm going with it, and why you should care. Or at least give you enough information to decide that in fact you don't ;) As I write more about the ideas I have I'll also be interested in feedback on whether what I'm doing makes sense, whether there are important scenarios it misses, and what I could do to make you interested in using it.

Ok, at this point I'm announcing vaporware. Sorry; I'm excited about this and don't have the discipline to wait until it really works. Besides, I've noticed that when I see an announcement of a new project, if it's provided as a fait accompli with lots of code and documentation, I'm not terribly likely to invest the time to delve through all that to learn what it's all about. If someone on one of the planets I'm subscribed to introduces their project over a series of blog posts, though, I find myself learning quite a lot about the project - and maybe wanting to use it - without feeling like I've invested any time or effort at all (after all, reading blogs is what you do to avoid putting effort into something else). If I cmeScribble can benefit from that effect in others, everyone wins.

Oh, and I guess I have to say that it's very Web 2.0. I'm not sure what that means except that all the cool kids are doing it these days. As far as I can understand of what that particular meaningless buzzword means, though, it does actually fit.


My compiler is psychic...

C:\projects\japitools\src>jikes -d .. net\wuffies\japi\*.java

Found 1 syntax error in "C:/projects/japitools/src/net/wuffies/japi/":

955. if (i != index && excps[i] is ClassType &&

*** Syntax: instanceof expected instead of this token


Talk about friendly error messages! How did it know? Does jikes encounter confused C# programmers often?

Previous PageNext Page