trailblazer and the curse of knowledge [07/17/2007 01:29:05]

Familiar code is fast code. No matter how clear or convoluted that code may be, our brains adapt as we write it, and we're able to hack away merrily at lightning speed.

But when we try to bring on a teammate, or even if we just put the code away for a few months and come back, the curse of knowledge becomes clear: the code we thought was good was merely familiar, and the details we thought were obvious now trip us up.

Even good software has this problem. For example, prototype.js works miracles for javascript, but until recently it was completely undocumented.

Now there's a really nice reference on the site, which is great for users, but if a new developer wanted to join the development team, you'll still have to dig into the code to find out how it all works, or lean on other developers for help.

The developer of a new piece of code and the adopter of an existing piece of code both face a learning curve.

The new developer has to learn how to frame and solving his immediate problem. This is something developers generally like to do.

The adopter has the much harder task of understanding someone else's code. Worse, there are generally multiple of libraries out there that might solve the immediate problem, and each of these likely solves other problems as well, so trying to find an existing solution often feels like a needle in the haystack problem.

Of course, solving the immediate problem is often not sufficient. There will be new problems tomorrow, and they will need to be solved too. It's not sufficient to be able to use the code, you want to be able to understand and modify it in the future.

That's why so many people (like me) roll their own frameworks. It may not be as good or polished as some other huge project, but we understand it because we watched it grow from the first line.

Knowing how to use something is very different from knowing how it works, or how to create it yourself.

Take bread, for example. Just looking at a loaf of bread, it's not obvious that bread was made of flour, yeast, water, and salt. Even knowing the ingredients isn't enough to make bread. You have to know how much to put in, how long to cook it, and in what order.

As developers, we constantly throw away our knowledge of history. We may preserve some rudimentary history in our changelogs and version control systems, but most of that knowledge just evaporates.

Right now, I'm trying to write a tutorial for some code I wrote back in October. I understand the code, and I have a rough idea what order I built it in, but even so, I find it very hard to organize the tutorial in a linear narrative. It's not so much difficult as it is intricate.

It's very rare to write code in order from the first line to the last. We may do this every once in a while for simple administrative scripts, but most of the time, we're hopping from line to line, moving things around, and building a complex network of ideas.

No matter how well we organize our code (or design documents, or test cases), it will always be a challenge to take that etwork of ideas and serialize it so that someone else can understand. That is because the linear history of what gets built first, second, third, and so on, is hardly ever aptured.

We don't even organize our plans that way. At least, in every professional project I've been a part of, plans were either created after the finished design document or (in XP) just sort of emerged from a deck of notecards.

In both cases, you wind up with a lot of needless work as the same knowledge gets invented over and over again.

For example, in a completed design, you have many pieces of code that support each other. Ideally, each class or function is independent and self-contained, but unfortunately, that's not always possible. There are dependencies, and so the order in which things are implemented makes a difference.

Of course, this gets even more complicated when the requirements change, which is one of the main things agile processes attempt to address. With XP in particular, the reworking happens at the code level (refactoring). Refactoring code is always going to be harder and slower than refactoring an idea (no matter how flat they draw your cost-of-change curve) but you end up with something you can execute rather than just a prettier set of boxes on a piece of paper.

In any case, what you always end up losing is a path from zero knowledge to an understanding of the complete system. Design docs and the code itself don't cut it, because they're nonlinear networks of ideas that have to be passed into the new guy's brain in some linear fashion -- because how many new ideas can a brain take in at once?

Pair programming helps tremendously here, because only one person needs to be able to navigate the code. Pairing is a fantastic way to bring someone up to speed. The new guy can still contribute because the old guy helps him get oriented, and at that point, you can both narrow your focus to getting the next test to pass.

Pairing is a general solution, but it does require having someone to pair with. If you're looking more at getting large numbers of people up to speed (for example with an open source release), or trying to preserve knowledge for yourself or other developers in the future, then of course pairing doesn't apply.

So another solution is to leave a trail: you build the plan as you gather requirements - the most important features go first. Then you plan how to implement each feature as you build your design. You do your refactoring up front in the design stage, always updating the plan. Ideally, the requirements, project plan, user manual, design, test cases, code, and comments would all be put into one big timeline that preserves the whole history.

Of course, it doesn't have to be the actual history, just a plausible history that moves the reader quickly from zero knowledge to working code. Months spent on an old design can be relegated to a footnote through the magic of retroactive continuity. That's why it's called trailblazer: trailblazing means marking a path through the terrain so that the people following the trail will have an easier time than you did.

Or, at least, that's the idea. Unfortunately, there isn't a tool out there that really supports this style of working yet, and I'm having to cobble together bits and pieces of a bunch of different tools. Even so, I'm making good progress with trailblazer. I'm learning how to apply it to existing codebases, and how it all fits in with outlining. Hopefully soon I'll have a javascript tutorial to show off, and I'll finally be able to show everyone what I'm talking about. :)