... indistinguishable from magic
effing the ineffable since 1977


Show All

Recent Posts



My blog is now running on cmScribe 5.0. This will mean nothing to most people reading this, but it's a big deal to me, because in 5.0 I managed to achieve a whole lot of things that have been goals of mine for cmScribe since before it had a name, and before anyone else who still works at NetReach now was even employed there. Except for Will but he cheated by leaving and coming back.

The upgrade process is fairly seamless but under the hood everything's changed. As well as running on .NET 2.0 (we could run on 3.0, but since the two are identical, why bother?) we're taking full advantage of the new capabilities of the 2.0 version of the framework: every ArrayList and Hashtable in our code has been replaced by a List<> or a Dictionary<> and our custom-coded nullable type wrappers have been replaced by the framework's Nullable<>. Instead of a kludged-together build process, we integrate nicely into Visual Studio's build process using msbuild. We've moved from SourceSafe to Team Foundation internally. And taken advantage of the fact that msbuild and tf are true managed applications by replacing all our hacky vbscript, perl and bat scripts for branching, upgrading and pushing with a single C# application.

The change that's most interesting to me is that in 5.0 we've managed to make cmScribe truly modular. We had something like modules before, but everything got compiled in one go into the same assembly, so there was no way to enforce rules about what code depended on what (except for me going over to people's cubes and saying OMGWTFBBQ at them). We also had a nasty habit of adding customizations to individual customer installations of cmScribe by changing files that were part of cmScribe itself; this was expedient but always caused issues when it came time to upgrade to a newer version.

Today, each module is built separately; dependencies are declared explicitly (and circular dependencies are impossible). And we've added - and enforced the use of - mechanisms to customize the behavior of cmScribe panels by adding extensions to them, rather than changing the original. The extensions typically live in a separate module from the panel being extended, so the base cmScribe modules are used unchanged.

The end result being that cmScribe modules are strictly separated from each other, and customer customizations are strictly separated from core cmScribe code.

Next up: trying to convince the Powers that Be that building a community of developers around some of the cmScribe core modules is possible and valuable and that liberty is the best way to achieve that :)


cmScribe guidelines

Apologies for blogging something that's only of interest to NetReach employees. But if I put it here I might actually get around to writing it - possibly even updating it - and at least some NetReach people will read it. Do's and Don'ts for writing cmScribe code:
  • Don't put anything end-user visible into an aspx page. Put it into a panel instead so that a template and styling can be added. If it needs to be on a separate page just add a configuration option in the admin control for the page ID.
  • Don't hardcode any user-visible text; always make it configurable in the admin control.
  • Don't put anything into Core without being very sure it belongs there; especially don't put anything into Core that has any dependencies outside of Core.
  • Put plenty of classes on user-visible elements to provide hooks for styling.
  • Don't put style into CSS files in the project if there's any chance a designer might want to override it. Create a Style in the database that the designer can edit instead.
  • Follow the conventions for administrative styling, like using class="nr-edit-table" and <th> tags.
In the past few weeks I've encountered and fixed all of these. So I thought writing them down might be a good idea... (Updated to add: Clearly I'm not immune to forgetting these rules - I just noticed that the text 'Show All' in the Categories panel over on the left there when a category is selected is not configurable anywhere, and I wrote that panel myself...)


The Unix philosophy of Lego bricks


As a child I spent an inordinate amount of time playing with Lego (was it just me or did anyone else invent names for the bricks? I had Long-ers, Square-ers, Tippees, Flat Tippees, Very Flat Tippees, Tipping Up Tippees...) The fact that a tubful of basic components could become the most fabulous spaceship I was capable of imagining had a huge influence on me and was probably a large factor in why I became a programmer.

I really think I'm onto something with cmeScribble but I've had a hard time coming up with a way to explain exactly what I think I'm onto. Part of that is because, obviously, I'm writing largely for an audience that isn't familiar with cmScribe, the philosophy it tries to embody, the strengths of its approach or the limitations we're starting to hit. Perhaps I should let my ten-year-old self explain by analogy.

We needed a powerful layout system so we came up with Lego.
Then we needed flexible hierarchical navigation so we built in Duplo.
Then we needed some place to put the content so we made a Rubik's Magic.
We built an eCommerce system out of Meccano.
We built a document repository out of Rubik's Snakes (those were great, weren't they?)
We made a general-purpose form builder out of MegaBloks.
We made a permissions system out of whatever that cool thing was that had all the translucent octagonal pieces with slots in the sides.

And every time we add a new feature we get a different type of construction toy to make it out of.

We've done a good job of making all these things work together. To display products in the nav we built an adapter that allowed Meccano to attach to Duplo. We built adapters so that octagons could be attached to anything. We built adapters to connect Meccano to Rubik's Magic. We built adapters and adapters and more adapters.

What we've ended up with is something that's extremely flexible and powerful, but in doing so we've solved the same problem lots of times - making one kind of toy talk to another, or making one toy appear as if it's a different kind of toy, or solving a problem for one kind of toy that was already solved for one of the others (notably, our Rubiks-Magic search is entirely separate from our Meccano search).

The idea behind cmeScribble is "what would happen if we just did it all with Lego[1]?"

The Lego philosophy and the Unix philosophy have a lot in common. In cmScribe we got the bit about "Do one thing and do it well" right, whether the one thing is "match a regular expression" or "be two by four, full height and red" or "display a product list". But just as important, if less expressable in a pithy slogan, is that the reason Lego and Unix work as well as they do is that every part has a common interface. In Unix it's the plain text file with each line representing some kind of value. In Lego it's the little round bump and the little bump-sized hole. In both cases the power comes from the fact that any piece's bumps can attach to any other piece's holes without having to create a custom Long-er-To-Very-Flat-Tippee-Adapter.

In the next thrilling installment: What do I propose as cmeScribble's little-round-bump-equivalent?

[1] Lego is, after all, the One True Construction Toy and MegaBloks can bite my small cuboidal plastic butt ;)


Are you mocking me?


Before I wrote a line of code for cmeScribble I decided that I'd code it the right way. Retrofitting "right-way-ness" onto an existing project is almost impossible, so I was determined not to miss the chance to build it in from day one.

The particular "right way" I had in mind was using Test-Driven Development, or TDD. For anyone unfamiliar with the TDD methodology, the idea is that before you write any code or fix any bug you first write a test or two to make sure the code works or the bug is fixed. The test should fail when you first run it, because the code doesn't work and the bug hasn't been fixed. Once you've written the test and verified that it fails, then you write the code to make it pass.

The advantages of TDD are well-documented elsewhere, but in a nutshell: writing the test forces you to know in advance exactly what you want the code to do, and also as you develop more and more code you have a library of tests to make sure you don't break anything.

I could easily see how to write tests for self-contained functions, but I couldn't figure out how to write tests for user interfaces or database-backed logic. Since cmeScribble is all about building user interfaces from database data, this posed something of a challenge. Maybe I can get away without testing the UI layer, but I can count features that won't need database data in some way on the fingers of Captain Hook's right hand. Okay, I thought, Google to the rescue... but searching for databases and TDD provided lots of people with similar issues and no straightforward answers.

A common suggestion was to use "mock objects". The idea is that you create a "fake" version of your database layer which returns plausible values without actually needing a database. This solves the problem but the downside is that you have to actually write the mock version of each database table, and since cmScribe has over a hundred tables at last count[1] I didn't relish that thought much either.

Facing a potential deal-breaker for something you don't want to compromise on tends to focus the mind, and eventually I hit on a solution. A full SQL database is hideously complex, but cmeScribble will be doing it's database access through nrdo (I'm biased, but I honestly don't see how people can work any other way...) and for 99% of uses, nrdo provides a fairly simple object model over the top of the database. Simple enough that if you don't care about persistence (which in a test situation is actually a bad thing) or performance, it's possible to implement the vast majority of it very simply in memory, as part of the nrdo runtime library and generated code, backed by simple List<T>s. A couple of days of hacking later and that's exactly what I've now checked into nrdo's CVS: If you use the C# 2.0 template and set the configuration setting "NrdoMock" to "True", all your code will run purely in memory, with a clean "database" for each run.

There are a few things missing - at a wild guess I'd say it covers 60% of what you'd want to do right now, and I should be able to get that up to about 90% without too much trouble by filling in things as I need them (after writing tests!). Most of the rest can't be handled automatically (because they'd require a full SQL engine in the mocking layer) but can be done with a little manual intervention (by the programmer providing a C# equivalent of the SQL clause in question; the SQL clause itself doesn't get tested but all the logic around it does).

The most satisfying part of the whole thing was watching it pass the tests I'd prepared in advance :)

[1] cmeScribble will have a lot less because of the "enterprise" features that aren't being included and also hopefully by unifying lots of things that were separate tables in cmScribe, but still.


cmeScribble - coming soon!


Last night I came up with the seed of a way to implement some things in cmScribe (the CMS that I work on as a day job and happens to run my website too) that I've wanted to do for ages but not known how - and get a bunch more cool features that I never even thought of before for free. That was the good news. The bad news was that the ideas are a radical departure from the way cmScribe works now and would require an almost complete rewrite - only a few pieces of code could be salvaged anything like intact. It's taken us three years of team development to get cmScribe to where it is now - which is, in my not-entirely-unbiased opinion, a pretty good place in lots of ways - so the idea of throwing all that away and starting over based on a mad programmer's wild and unproven blue-sky ideas was probably not something I could sell to the boss ;)

(Basically, the entire architecture of cmScribe today revolves around building templates and pages out of building blocks that we call panels. The idea I had leaves no concept of a panel left at all, and templates and pages don't have any independent existence but just become particular ways of combining lower-level ideas...)

But I think I found a way to turn the bad news to my - and NetReach's, and maybe your - advantage. Because it does mean a complete rewrite, there's no need to use any existing cmScribe code to implement it. So I suggested to my boss that I'd be happy to work on this in my own time as a hobby project / technology testbed, and wouldn't it be cool to do it as an Open Source project? As a hobby project it wouldn't even attempt to provide the "enterprise" features that cmScribe does, but if it successfully proves the concept, it would also be possible to use it to rebuild those enterprise features and get a next-generation cmScribe. My boss agreed.

So without further ado, I give you...

Well, actually, I don't, because all cmeScribble is right now is a logo, a pronunciation (see-me-scribble!) and some partly-formed ideas. There are parts of the design that are defined as being done "by magic". And while some incredibly rich and powerful scenarios are enabled by this architecture, I'm a little fuzzy on how to implement obscure things like, say, pages.

When I have some code (I've started, but I don't want to release until it can do something that at least gives a hint of the power to come) I'll put it online for people to try out. In the meantime I'll blog a bit about the process by which I arrived at the new design, where I think I'm going with it, and why you should care. Or at least give you enough information to decide that in fact you don't ;) As I write more about the ideas I have I'll also be interested in feedback on whether what I'm doing makes sense, whether there are important scenarios it misses, and what I could do to make you interested in using it.

Ok, at this point I'm announcing vaporware. Sorry; I'm excited about this and don't have the discipline to wait until it really works. Besides, I've noticed that when I see an announcement of a new project, if it's provided as a fait accompli with lots of code and documentation, I'm not terribly likely to invest the time to delve through all that to learn what it's all about. If someone on one of the planets I'm subscribed to introduces their project over a series of blog posts, though, I find myself learning quite a lot about the project - and maybe wanting to use it - without feeling like I've invested any time or effort at all (after all, reading blogs is what you do to avoid putting effort into something else). If I cmeScribble can benefit from that effect in others, everyone wins.

Oh, and I guess I have to say that it's very Web 2.0. I'm not sure what that means except that all the cool kids are doing it these days. As far as I can understand of what that particular meaningless buzzword means, though, it does actually fit.

Next Page