The past couple weeks have been one of the most stressful periods in my life. Sort of an out of the frying pan and into the fire situation as I finally recovered from the all-consuming client work period, where I had plenty of money and no time, to struggling to make ends meet in my business.
The economic problems in the world have not been kind to cornerhost. Lots of people are suddenly finding personal web hosting to be a luxury. Combine the general trend with poor support when I was swamped with client work, and escalating performance problems during the early part of the year (which persisted until I did a huge upgrade of all my servers), and things are looking pretty grim.
I missed two or three paychecks in a row there, and even wound up unable to afford my ADD meds for a couple weeks, which I basically spent in a complete fog.
But... I still have a roof over my head, and I still have a trick or two up my sleeve.
My hard drives are packed full of half-finished projects, many of which could easily be turned into commercial products if I could just finish them. Of course, many of those are web apps, and my decision to abandon my 10-year-labor-of-love web framework in favor of rails removed one of the major constraints on my performance, for the price of a learning curve that isn't even that steep after all.
Even now, though, back on Vyvanse and armed with a clear goal and deadline (for the bodyblog.com redesign), I'm struggling to stay on task and make the best use of my time.
I've been working steadily on The Flake Effect since November, to the point where I'm going to have to start looking for a system to manage and organize it all. I actually have a pretty clear idea of what an attention management software system would look like. It would combine sensors (such as checking email) with schedules, logical inference, metrics, planning algorithms, time constraints, chronobiology, and even a little NLP. Basically, it would create an immersive mental environment that turned productivity into an addicting video game.
I keep wanting to make the dang thing but it's too big and too complex to build, so I wind up building a little app here, a little script there. Plus there's the whole bootstrapping problem - my efforts to build a productivity tool are hampered by my poor productivity.
Time and time again, I find myself wishing that I could just leave my whole system running, and that I could tie pieces together more easily. Like, it's easy to make a daily checklist in a spreadhseet, but to hook that up to a timer requires all kinds of nasty cross-process communication. So I have a little bit that's in a web app, a little bit on the command line, another piece that's a custom GUI app, and none of the pieces ever talk to each other.
Things would be so much easier if I could run all these apps in a single process, modify them while they're running, and connect bits and pieces of them together on the fly.
The thing is, I don't have to sit around wishing for a development system like that, because it already exists. It's called Squeak.
I've resisted using squeak for basically the same two reasons I resisted rails: culture shock and the learning curve.
Squeak uses its own cartoony looking UI, has a completely alien concept of source code control (all the source code is inside the running squeak instance, rather than stored on separate files on disk), and is basically its own self-contained little world. Those are just a couple examples of culture shock.
By learning curve, I just mean that it's frustrating to have to look up every little concept, and then deal with the surprise when they don't have a word for what you want to say and you have to find a completely different way of expressing it. It really is like trying to express yourself a foreign language: you know exactly what you want to say, but you have to go word by word with the translating dictionary in hand, and even then you wind up with a jumbled patois of the grammar from the language you know and the vocabulary from the language you're trying to speak.
Prior to rails (and maybe drupal on the previous project), I always enjoyed reading about new and esoteric programming languages, and even tinkering with toy examples, but when it comes time to make a new application, I'm generally tempted to stick with what I know best.
Having pushed through the culture shock and learning curves a couple times, now, though, I'm starting to see the error of my ways.
In fact, it can be sort of fun. Yesterday, for example, I found myself reaching for a familiar python idiom, ran into some minor ruby culture shock, and wound up learning a different way to think about the issue, and a slightly better concept of object-oriented design.
I won't go into too much detail, but my goal was to map a symbol to a member of an object, so if you gave it a Person object and a variable field="name", it would return person.name. I python you say getattr(instance, field) but in ruby you say instance.method(field).call. There are like three or four huge conceptual differences highlighted in just those two lines of code, but in a sense it comes down to culture: in python, what's mine is yours, so you're free to take what you want, but in ruby you have to ask me nicely if I have what you want, then ask if you can borrow it, and then if I say yes, I give it to you in a little box and you have to take it out yourself.
Who knows? Maybe that reflects the difference between the Dutch approach to manners and the Japanese. :)
Squeak reflects more American ideas. It's a fast paced melting pot with a lot of innovative ideas, sometimes thrown together with duct tape. It gets the job done its own way and it doesn't care too much about appearances.
With most programming languages I feel a bit like a general organizing my troops and planning my attack. There's an overall strategy for where I'm going and many small tactical battles with the problem.
With squeak, even with the brief experience I've had with it, I feel more like a mad genius tinkering away on my latest invention in a workshop full of tools and gadgets. If I need a part, I can just cannibalize some other contraption, and if I need to hook two things together, then I've got all kinds of wires and duct tape to make that happen.
Or in other words, squeak jives well with my ADD.
Obviously, learning a new programming language or two isn't enough to turn my business around, but between rails and squeak, I really think I can remove some of the major constraints that were holding back the flow of ideas from my brain to the marketplace.
If my gamble pays off, then I can finally start delivering on my huge backlog of half-baked website and software ideas. Even if none of them make any money on their own, they might at least direct enough attention my way to get my little hosting business back on its feet.
So... Here goes nothing!
My friend Zach has this old ASP site over at bodyblog.com. Back in the day, it was a pretty active community of bodybuilders and regular people just getting in shape.
A couple months ago, I agreed to help him dust it off and get it working again, and since the code was pretty bad and ASP is a dead technology (replaced by ASP.net), I advised him to port it to a newer technology.
Then the question was, which one? I've been tinkering with my own little python web framework for years now, and while I'm proud of it, I couldn't in good conscience advise Zach to build his business around it.
If you're building a business you want something approaching an industry standard, because the more people working on and with a technology, the more developers you have available to hire, and the more tools they have to help them work faster.
Java and ASP.net were certainly viable options, but given my own areas of expertise, and the fact that I'd be doing most of the work, it came down to three choices:
Of the three, I had extensive experience with drupal, no experience with django (but over a decade of python experience) and only a vague familiarity with rails (having worked through a book a couple years ago and not touching it since)
It wasn't much of a contest.
Drupal is impressive, but it's a case of bright people making the best of a pretty clunky technology (PHP).
Django is... Well, it's written in python, lots of people seem to like it, and I can't think of one bad thing to say about it. Django is OK.
Rails, on the other hand... Rails sucks. In fact, rails sucks hard. The Railiens make a big fuss about favoring convention over configuration, and of course they picked the most bone-headed, moronic conventions imaginable, like coupling data objects to the database, using pluralized table names, and... and...
And none of those choices make the slightest bit of difference in the long run.
Rails is something people either love or hate with a passion. It has a personality. It's opinionated software, as they say, and it turns out that was a pretty brilliant marketing move.
A year or two when I worked through that rails book, the system I saw left me wondering "so what?" But what rails has going for it is a network effect. It drew in this core of very passionate users (especially corporate users) and they contributed their passion, and their resources, and it just kept growing.
We picked rails because it's reached its tipping point. The industry is behind it, there's a huge freelance developer population, and there's plenty of really great tool support.
Once I got past the culture shock and accepted that the conventions are just a matter of taste, and once I got past all the hassles of installing and configuring the dang thing (I finally gave up trying to get ruby talking to MySQL on my home windows box and just moved everything over to linux) I noticed that I was actually getting a hell of a lot done in not much time at all.
For the longest time, I held on to this idea that I could become some kind of tech leader. Some kind of Prometheus complex, maybe: part of me really wanted to be the one stealing fire from the gods and leading everyone into enlightenment.
But for the past few months... I dunno. I just don't care about that anymore. I don't feel like I have to leave my mark on the world. At least, not as a web framework developer.
So yeah. Long story short: rails won.
python that looks like haskell:
from arlo import declare, _
exec declare('sig let Ord a where y xs x ')
qsort = [
sig. qsort == Ord.a >> [a] >> [a] ,
let. qsort ( [] ) == [] ,
let. qsort ( _[x:xs] ) == ( _.qsort.lesser + [x] + _.qsort.greater ,
where (
_.lesser == [ y | y << xs , y < x ],
_.greater == [ y | y << xs , y >= x ])) ]
The first line imports a function (declare) and an object (_). from the module arlo.
The second line calls declare to convert a string of identifiers into a string of python variable declarations, and then runs the generated code.
The remaining lines create a list of three objects, and assign it to a new variable called qsort.
The point of this program isn't to emulate haskell in python, or even to implement quicksort, but rather to demonstrate how arlo lets you hack the python syntax to express interesting ways.
expressions that look like statements:
from arlo import declare, _
exec declare("CLASS YIELD RETURN SET DEF SELF INIT")
print repr(
CLASS. MyClass ( _.object ) [
DEF. INIT (SELF, _.name ) [
SET. SELF.name == _.name ],
DEF. hello (SELF, _.name) [
YIELD. _("hello %s") % SELF.name ]])
The first two lines are the same as before, except for the names being declared.
The rest of the program builds up a single object and then prints out a string representation of that object. It looks like this (except scrunched together on one long line):
_.CLASS.MyClass(_.object)['_.DEF.INIT(_.SELF,_.name)
['(_.SET.SELF.name == _.name)'],_.DEF.hello(_.SELF,
_.name)["(_.YIELD._(_('hello %s')) % _.SELF.name)"]']
What's interesting is that if you loaded that output into a python string, evaluated it with python's eval() function, and then printed the resulting representation, you'd get that same output
again.
the reification rule:
# for any arlo.Expr object s:
assert repr(eval(s)) == s
In other words, arlo lets you create
python expressions that model the syntax used to create them.
Why would you want to do this? Well, as you can see from the examples above, by stripping the meaning from the various python operators, you can create your own mini-languages on the fly, without needing to write your own parser.
arlo at the prompt:
You can also combine these expression objects directly at the python prompt, and switch back and forth between the expression language and pure python. For example, notice how python immediately evaluates 2 + 3 in the fourth line below, whereas the other addition operations get quoted into the expression object.
>>> x = _.x
>>> x
_.x
>>> x + 1
(_.x + _(1))
>>> x * ( 2 + 3 ) + 2
((_.x * _(5)) + _(2))
>>> x.y() + _.z
(_.x.y() + _.z)
Another way of saying this is that arlo
is a generic combinator library for building expressions.
Combinator libraries are common in functional
languages like haskell, but they're not exactly
new to python. For example, pyparsing uses the approach to build parsers, while Stan overrides method invocation (x(y)) and subscripting (x[y]) to provide a pure-python syntax for building XML documents,
and the SQLAlchemy expression language uses the combinator concept for building SQL queries.
What is new (at least as far as I know) is the idea of creating a generic combinator library for expressions.
We've already seen that calling repr() on
these objects produces the python syntax necessary to recreate them. But if you call str(), you get the equivalent python expression, which you can then pass to eval():
Sort of like a lambda:
>>> y = _.m * _.x + _.b
>>> y # repr()
((_.m * _.x) + _.b)
>>> print y # str()
((m * x) + b)
>>> m, x, b = 1, 2, 3
>>> eval(str(y))
5
>>> x = 4
>>> eval(str(y))
7
As you can see, you can use arlo and eval() as an alternative to defining named functions or using lambda.
About two years ago, I wrote a post called "what python looks like naked" that demonstrated how you could remove all of python's control structures from a program and replace them with functions and delaying every expression by wrapping it in a lambda.
If you can replace python's control structures, that
means you can create your own control structures. For example, you could use the technique to add a prolog-style inference engine to python. Then, in addition to simple propositional logic (if/elif/else and the boolean operators), your program could make choices based on logical deduction from a set of facts and rules.
Of course you can implement these things in python
already, or call foreign libraries, but it would be nice
to have these things as first-order objects in the language, and that's what a reification technique like wrapping everything in lambda gives you.
The only problem is, lambdafied python is an ugly, ugly mess.
The combinator appproach goes a long way to solving the syntax problem, and with arlo, python developers can now use a generic syntax for building combinator libraries.
Anyway, this is an idea I've been kicking around for a long time, and now that it's working, I'd love to hear what other folks have to say about it.
try it out!
(it's open source under a python-style license)
You can browse or download the code from the Trac links below:
You might also want to look at wherewolf, a simple language
for building expressions for matching against tables of data. Thee expressions can either be evaluated directly in python for querying against data in memory, or compiled down to SQL WHERE clauses (like a very tiny version of SQLAlchemy).
I've been sick in bed with the flu all week. Last week I was in Texas with my family. Given past history, my world should basically be upside down right now.
I do have a larger-than-normal backlog of stuff to do right now, but I actually feel like I'm on track. Once I was finally able to breathe and sit up this morning, I got to work, and I expect to be caught up completely within the next day or two.
That's a big change. In years past, the backlog (especially with email) would persist for days or even weeks.
Forcing myself to sit down and write The Flake Effect last month was one of the smartest things I've ever done. I didn't really come up with a book so much as a long, rambling journal focused on productivity. But I covered a lot of ground, and I verbalized, clarified, or outright solved a bunch of problems that had been holding me back for years.
It's going to be a long time before I have a second draft worthy of showing off to anyone, but I do want to share some of the key insights I gained, especially in the context of planning the next year.
First off, I hit the official nanowrimo wordcount in the first week. I thought I was aiming for 100,000 words but it turns out it was only 50,000. It's not really a novel. More like narrative nonfiction.
Second, part of the way I was attempting to handle "characters" was to talk about ideas with my friends in chat and upload the logs (I stripped out the repeating names from my wordcount totals). That was really the first time I tried to explain my ideas to my friends directly... At least in a long time, and when they weren't explicitly asking for help with something. It made for some interesting conversations and I think it helped me understand my friends a little better too.
Did I actually learn anything new about productivity and getting things done?
I feel like I did, but right now it's hard to explain concisely. I certainly compiled a bunch of techniques, like:
I also am going to have quite a bit to say about attention management in general, especially with regards to the idea that attention is fractal.
The fractal metaphor is only going to go so far, but there are a lot of parallels between the questions "how long is a coastline?" and "how much attention can you pay to something in an hour?" In both cases, the answer keeps increasing the closer you look.
In other words, attention isn't really measured in time directly but something like details per second, square-inch-seconds of brain activity, or maybe flops if you're talking about a cpu. We don't really have tools to measure it outside of MRI lab (though several portable consumer EEG machines are close to market, and they come a little closer), but the point is attention management and time management are two totally different things.
Also, you can't make more time but you can create more attention through the miracle of neuroplasticity.
But anyway... There'll probably be a whole chapter on that, so no point rewriting it here.
The other major insight that came to me is the distinction between two types of problems. This kind of fits in with the fractal idea, though at the moment, the idea is still very fuzzy in my mind. In fact, it may not even be a distinction between problems but between problem solving methods.
When I say "deep" I'm picturing a deep canyon that has been carved by millions of years of water flowing over the same course. An example would be mastering a specific skill like aikido or playing poker or painting a picture, or developing a framework to solve a particular programming problem. Or building a fully automated business.
Or for example, I sometimes make the distinction between writing to explain things to myself. It doesn't matter if I ramble and take 10 pages to get there because the only thing that matters is that the problem winds up solved in my own head. It's like casting a wide net and hoping to catch something good.
But if I want to explain the idea to someone in particular, I need to go back and harvest that idea. Rewriting it, simplifying the mental path for my audience and filling in the intuitive leaps.
Anyway, I'm a whole lot better at the "wide" approach than at the "deep" approach. I wind up knowing a little bit about a lot of different things, but I don't have any real expertise.
That means I can look at a new problem from a bazillion different points of view, and come up with a solution rather quickly, but then I have trouble actually implementing a solution. (Which, when combined with a tendency to follow lots of little tangents means I wind up with a whole slew of unimplemented ideas - the very definition of the flake effect.)
Anyway, so I see now that I could benefit tremendously from working on the "deep" habits - daily practice, refinement, continuous improvement, etc.
Well, first of all, I want to put the ideas into practice in my own life. I've made a start on that, but these are things that need to be habits, not just one-time exercises.
Since this is "deep" stuff and I suck at that, there will probably be a lot more to write along those lines. In fact, I may wind up writing enough for a second book before I figure out how to tame the first book.
But I do want to tame the book. Go back and refactor the ideas. Create an actual organizing structure.
And then what? I don't know. Blog about it. Maybe sell it as an ebook. Maybe work with some clients or start an online community so I'm sure I have a system that works for other people too.
I feel like I've got a complete idea now, and it could be a book as-is, but I don't really know if it's useful to anyone besides me, and I also don't know if it'll continue to be useful to me in the months ahead without further improvement. (I need to test it against a requisite variety of situations, as my friend Leslie might say.)
Anyway, my immediate goal is to use the tools I have now to get caught up on my various responsibilities, and then start working on some goals and plans for next year.
Well, I'm finally doing it.
I didn't want to say anything because I didn't really know whether or not I'd I'd follow through, and I sort of felt guilty about spending time on this when there's so much else I'm supposed to be doing for other people.
But it turns out I am following through (in fact, I'm way ahead of the already aggressive schedule I set for myself)...
And also, it's helping me sort through the problems that have kept me from delivering for my clients, customers, and even myself.
Long story short: I signed up for NaNoWriMo, and I'm finally writing The Flake Effect - the book about getting things done that I promised myself I'd write some day, whenever I figured out how to get things done.
Since I don't really know how to get things done (despite reading countless books and struggling with the problem for years), that's the central conflict of the book. It's about problem solving, bootstrapping, neuro-linguistic programming, creativity, the theory of constraints, running a business, and basically all the stuff I normally write about anyway. :)
I posted the rough first chapter. It's 10,000 words and full of my usual rambling. It doesn't really fit with the rest of what I've written, and I'm not planning on posting any more text until it's done, but you can read it and my little status blog on the writing effort over at flakeeffect.com.
"There's way too much information to decode the Matrix. You get used to it, though. Your brain does the translating. I don't even see the code. All I see is blonde, brunette, redhead. Hey uh, you want a drink?"
--Cipher
This is the kind of thing my brain comes up with when I forget to take my vyvanse.
I will live each day cheerfully and with a sense of adventure...
Thus begins my personal mission statement - the document designed to guide my every decision.
And yet... I have to admit I have not been very cheerful lately.
If you (like me) [ heh heh heh ] need a reminder of what it means to be cheerful, please direct your attention to the following lecture by rock star and particle physicist, Dr. Brian Cox:
I'm just rambling here. You can tell by the way there's no little hand drawn picture on this post.
Which is actually funny, because what I'm rambling about is Quality, which is largely a byproduct of having a Process (love the capital letters) and the real gist of this post is that by sticking to your Process your quality goes up and your headaches go down.
So I've been doing client work, and part of what I'm doing is trying to help fix the broken development process in an otherwise awesome company.
I've been working long hours. I have the dark circles under my eyes. It's 10:47pm right now and there's stuff I ought to be doing for the morning and it's probably not going to get done.
Basically, a product got launched before there was any e-commerce software to back it up. The shopping cart wasn't finished. The code to talk to the fulfillment center hadn't been written. There were no automated tests to be found. I spent several hours the first night manually diffing and merging files into a new CVS repository because there were multiple people working on the site and they didn't even have version control set up.
No... Actually it was worse than that. The company has a version control system but the developers in charge refused to use it.
Anyway, I've been doing a lot of inter-department diplomacy the past couple weeks. Everything's in subversion now, but we're still putting out fires on a daily basis.
I found out today I started one of those fires. I rushed some code out the door because we were in such a hurry and it had to be done right now and so I wrote this really fancy SQL query.
The first time it came back, I found out it was grouped wrong, so I fixed that. Only it was a quick fix because I was overwhelmed with all the other fire-fighting work.
That was a week ago. Today it comes back again, after people have acted on the data, and it turns out there was a massive bug in my SQL:
select if(some_string, use_this, else_this) ...
Do you see the bug? I sure didn't. And I read the if() docs right before I wrote the query.
Turns out that strings are evaluated as false, regardless of their content. (In python and many other languages, the empty string "" is false and all other strings are true)... So basically the "use_this" value never got used.
I'm not ashamed of that mistake. The dang query still looks right to me.
But I am ashamed that I let myself get caught up in the frenzy and shipped mission-critical code with only a cursory visual inspection.
All along, I've been thinking: Let's just get through this launch, and then we can start doing things right.
But after today?
After today, I'm honestly worried that these problems are just going to keep on going. We're so busy fighting fires that we're not doing anything to prevent new ones.
Hell, I'm already worried about the next launch slipping. Everyone seems to think we have plenty of time. But I think we're three weeks late unless we make some massive changes NOW, and this is a project we haven't even started yet.
What's really funny is that on everything else - like the ability to change processes and fix communications problems - I've been scoffed at for being overly optimistic. :)
Anyway, I made the decision today that this has to end. From now on, we write unit tests for every line of production code we create.
That is absolutely the right decision.
And yet I felt like a complete jerk today, telling the guy that just worked overtime 4 days in a row on a poorly spec'd app we absolutely had to have right now that no, we can't ship this after all until we go back and build unit tests.
So this is management, huh?
Actually, I'm having fun.
It's stressful and time consuming, but everyone's been remarkably understanding that it's the process at fault, not individual people. It really is a neat company.
Anyway, I just needed to vent here... I guess I cheered myself up. Optimism comes in handy sometimes. :)
Back to work.
Guys, I'm really trying.
Yeah, I know you've already solved my problem.
Yeah, I know it's more work to write my own code than to use yours.
But the thing is... Your documentation sucks.
It may be more work to write my own code, but it's far less frustrating than reading your docs.
Let's name some names here.
Hey Glyph Lefkowitz! The twisted documentation sucks!
Hey Ian Bicking! The paste documentation sucks!
Hey Richard Jones! The roundup documentation sucks!
Hey Phil Eby! I honestly don't know whether The EasyInstall documentation sucks or not because I've usually given up by the time the page loads. is pretty darn good. But your page load time sucks!
Hey Armin Rigo! The PyPy documentation sucks!
Hey Kay Schluehr! The EasyExtend documentation sucks!
You guys are responsible for some of the coolest, smartest python projects out there. I'm sure they're all worth the learning curve... If only I had time to decipher your documentation.
I'm not unreasonable. I ditched the home-grown template system that I used for years in favor of genshi. Genshi solved my problems, and it's got fantastic documentation.
In fact, other than PyPy and EasyExtend, I use all the above tools in my company - some of it on a daily basis.
But the simplest, one-line tasks take me hours and hours to accomplish (or just don't get done) because I can't understand your documentation.
I would be happy to sponsor a rewrite. You're wasting my time and costing me money so a rewrite would almost certainly have a positive ROI. Heck, I'd be happy to rewrite the docs myself.
The only problem is... I don't know enough about your apps to explain it to anyone, and I don't know how to obtain that knowledge.
Can you guess why?
It's because your documentation sucks. :/
Here's a screenshot of the client site I've been working on:
The site is American Program Bureau. They manage the speaking engagements for lots of big name celebrities. So if you want Dan Rather or Alan Alda to drop by your next meeting, they're the guys to call.
I wish I could take credit for the design, but I didn't have much to do with that. I built the drupal backend, cleaned up and extended some actionscript for the various flash widgets, and got their old data into the system.
Anyway, we just brought it live, so I figured I'd share. :)