Update: Light Table reached its funding goal, and Chris promises to release the source as soon as is practical (see the comments) - everybody wins!

I love the principles behind Light Table, Chris Granger’s excellent IDE concept based on Bret Victor’s incredible vision. It’s up for funding on Kickstarter now, but it might not make it’s $200k goal:

Honestly? I’ll be happy if it doesn’t get funded.

So is it Light Table you don’t like, or is it personal?

Are you kidding? I love Light Table and I think the world will be a better place for it. I don’t know Chris, but I’ve read some of his code and have a deep respect for him and his work.

Why don’t you want it to get funded, then?

I’ll be happy if Light Table isn’t funded because Kickstarter feels like the wrong thing to do. Chris writes:

I haven’t thought about everything - and I won’t be able to… Light Table is meant to be a platform and it will be open source for that very reason… This is the power of a community full of builders… In the end, the only way to move us forward is for the community to rally behind something.

— Chris Granger, On Concepts and Realities

I couldn’t have said it better myself. To me, this makes the choice for how to proceed with the project crystal clear, but Chris seems to have chosen poorly. There are two options:

Option 1: TEN MILLION DOLLARS!

As the concept video reached 80k hits and the blog post 200k, I’d have been thinking:

Wow, two hundred thousand hits, that’s like - if every one of those bought a licence for $50 then I’d have TEN MILLION DOLLARS and a dream job and would never have any worries ever again! Hooray!

— What I’d have thought, if I were in Chris Granger’s position

There’s nothing intrinsically wrong with this. Clearly there’s a need for a product like this, and if a (finished) version of Light Table for Python existed today I’d drop $100 on it without even blinking. Perhaps even $500 for C++. And I’d be happy to do so.

Chris has chosen this route - trying to build a profitable business - and to his credit he’s also chosen to open source the core of Light Table, albeit in May 2013 after its release. And you still have to pay for a license.

Which brings us to his mistake.

Option 2: Change the World

Instead of putting up a kickstarter page, Chris could have put his source on GitHub, written about the principles he hopes the IDE will enshrine and started accepting patches. It would have instantly become incredibly active; he’d have his work cut out reviewing and accepting pull requests, discussing UI design and principles and making his vision real.

Perhaps it would have been valuable enough to the community for him to go full-time, in which case he could have opened a kickstarter to fund himself as the leader of the open source project. Set a low goal (one month’s worth) and then see what the community wants to pay him to do. There are other options too, such as looking for corporate sponsors, or just doing what he can in his spare time and delegating the rest.

The end result is less certainty for Chris, but much more certainty for everybody else.

What’s Wrong with Option 1

Compare these two projects, for a moment:

1. Nothing happens for a year while two to three people work together. People get to play with beta release and give suggestions, but every single line of code comes from that handful of minds. Most of the world has to wait for 12 months and then they get to buy a Clojure IDE. Yay.
2. A veritable explosion of activity as everybody forks the project, tries to get it running for their own favourite language, adds the features they need to make it useful in their day-to-day life. From day one, whoever you are you can find a fork adding support for your language, the better of which are progressively pulled back into Chris’ main repository. Conservatively, hundreds of developers working together. What will this look like after 12 months? Will there be Python support? Ruby support? Java? Integration with other tools? Vim bindings? Emacs? Themes? Yes, yes, yes, and more.

Clearly option #2 is going to be better, both immediately and after 12 months. Trying to develop a community-style project like this with a small team and closed source is a ridiculous mismatch.

Living Costs Money

There are ways for Chris to get paid to work full-time on option #2, but they ask for a lot of trust - a leap of faith that a community will develop that’s willing to support him. It means taking the chance that the project won’t need him and that it will get along just fine without his full-time attention.

What about later on? Chris and team might get rich selling option #1 licenses for years to come; how can they do that if everything is free and open source with an active, vibrant community?

Here’s the choice reframed: either your time on the project is valuable enough for other people to give you money to do it, or it isn’t. You can open source the project and find out directly, or you can attempt to distort your value by creating a false scarcity, only licensing it to people who pay you.

I don’t know if I could make this leap of faith. It’s still the right thing to do.

Your Choice

Do you really want to extract money from people to work on something the world doesn’t need you to? If not, what are you afraid of?

If you believe the world needs your ideas, if you want to spend some of the few years given to you bringing them to life, then do whatever you can to share them and make these concepts real. In this case, it’s clearly to open source Light Table and trust that the future will bring good things.

Don’t be afraid.

Believe.

Teamwork is a tricky business. I write software for a living and teamwork amongst programmers tends to be defined as splitting up the work such as to minimize the need for any actual teamwork or, indeed, communication of any sort for the remainder of the task.

As a direct result of this, there’s often a short, socially-charged moment in any meeting in which the ‘team’ work is divided up into my work and your work, aka your problem. I’m lead to believe this isn’t unique to programmers.

There are so many little games played here as we jockey for social position, each wanting to get an interesting or easy piece of the work and to avoid the awful, blatantly difficult and unpleasant part that nobody wants to do. Don’t let it be me, we pray.

Those with social clout in the group use it now to quickly snap up the high-prestige-yet-surprisingly-easy work. Occasionally someone without enough influence to snatch a safe piece of work for themselves will throw a victim to the wolves. Well, Jane has always been good at random-terrible-thing, haven’t you Jane, so I guess it makes sense if I do unglamorous-but-safe-task.

Poor Jane.

And like this, Jane - perhaps the youngest, newest, or simply least-liked - ends up doing the work nobody else wants to do:

1. Legacy work. Cleaning up a mess somebody else made long ago that has been rotting away for years, becoming so bad that it can’t be ignored any longer and yet is extremely likely to devour the soul of anyone who disturbs it. 
2. Ill-defined work. Nobody’s thought through exactly what’s going to be involved, but everybody says it should be straightforward because they’re sure that by the time it becomes clear it’s actually giant can of writhing, flesh-eating worms that it won’t be their problem any more.
3. Restructuring work. Everyone agrees it needs to be done, but customers will never see it, management will never understand it and your colleagues will complain loudly if you get in the way of their work while doing it. Afterwards, everyone will say the old way was better. 

In other words, nobody wants to do work that’s difficult, risky and thankless. If there was a map of the landscape of work, this bit would be the vague, sketchy part marked simply with here be dragons, where the unfortunate are sent to their fate. I’ve played the don’t-get-burned game and even got rather good at it, which is easier when you realize some participants don’t even know that they’re playing, but that’s irrelevant right now.

You see, I don’t want you to play this game. Yes, you can wheedle your way into a position of some authority by building up your little corner, looking good to your superiors while betraying potential rivals with each step, always making sure someone else gets sent to the flames. And when you reach power, that’s the person you’ll be and the sort of people you’ll be working with.

There is another way. When deadlines are looming and the project is in trouble, people don’t turn to little empire-builders for help and leadership no matter how shiny they are. We don’t want someone to delegate away the dragons, we want a dragon-slayer. In tough times we follow a dragon-slayer because we know he won’t abandon us to our fate. We trust him because he’s proven his worth time after time while others tried to get away. And we love a dragon-slayer, because he makes it work instead of parceling out blame to the weak. A dragon-slayer has a freedom and authority that no amount of empire-building will ever match. That’s who I want you to be.

So when you find yourself in one of these nervous meetings where the work is being divided up, look for the part that everybody’s afraid of being given. You need to stand up and take that before anyone has a chance to react; take it and make it yours. Yes, it’ll be hard and risky and thankless, and sometimes you’ll get burned, but your colleagues will remember that if you hadn’t stood up it could have been them, and they will respect you when you stand up a second time, and a third, and a fourth.

Eventually, you’ll stop getting burned and start emerging victorious, because practice makes perfect and the experience you’re getting is the very best there is - always working on difficult, risky, problems instead of avoiding them and passing them off to the weak; those least equipped to deal with them. One day, you will begin to succeed and nothing will ever stop you again.

But that’s in the future and this is the here and now. Look around you and you’ll see what it is out there that nobody else dares to face. If you want to become a dragon-slayer then you’d better get started - and there’s only one way to do it.

You have to fight the dragon.

Recently I started optimizing my workflow with various scripts and tools. I measured my performance and found it increased a lot.

Some people asked to see stats behind the story so here they are, calculated retrospectively from log files for 5 months before the change and 5 months after it.

4x more productive overnight

I’ve been tracking 3 easy-to-measure metrics:

1. Bugs fixed
2. Change sets committed
3. Number of functions changed

For your viewing pleasure and mine, I’ve tweaked my script to render graphs with the GChartWrapper python module:

I started trying to improve my own performance sometime in mid-January; unfortunately I didn’t keep track of the exact date.

The data are scaled to show bugs fixed per man-month rather than calendar month; from February onward I’ve been in parental leave (Germany’s extremely generous Elternzeit to be precise) and have been working exactly half-time at 18 hours a week.

As a rough measure, let’s compare my performance in Aug-Dec to that in Jan-May, ignoring the fact that I started this scheme mid-way through January and not at the start of the month.

Performance boost: 55.4 / 9.4 = 5.9 times more productive

A changeset can be anything from a bug that was too minor to both putting through the bug-tracking system (such as a message box typo) or a little tweak or extra feature added in passing. Static analysis tends to throw up lots of small fixes that I address with direct commits, for example.

Performance boost: 100.4 / 28.0 = 3.6 times more productive

I started tracking functions changed just out of curiosity; the numbers of tickets fixed and changes committed are open to various levels of manipulation - I thought the total number of functions touched by a change would be an interesting check to make sure I wasn’t just solving all the easiest tickets or committing lots of tiny, pointless changes.

Performance boost: 344.8 / 85.6 = 4.03 times more productive

Hey, is that awesome?

Nothing about the above measurements or calculations is perfect; simply ignoring December artificially depresses the performance boost, as does treating all of January as the post-boost phase. On the other hand it seems incredibly unlikely that working half-time is exactly half as productive as working full-time.

We could argue that the email and communications burden remains the same when working part-time, so we’d expect less the time to be available for productive efforts. Or on the other hand that most of a full working day is wasted anyway.

We could argue a lot of things, but none of these concerns can account for a 400% increase in productivity.

Not only that, but the number of bugs fixes (5.9 times increase) - is a metric of real value to our company. Generally our software director chooses the set of bugs to be fixed for each release and gives them priorities, which means all those bugs had to be fixed and I wasn’t at liberty to pick all the easy ones.

I do like fixing the easy ones, though.

Whether my measured productivity has doubled or increased ten-fold is really neither here nor there; even just doubling your productivity by any set of measurements over a sustained period of five months is incredible.

I’ve also been a lot happier, which is like winning twice.

Wait! There are 3 reasons not to do this!

“I changed my editor bindings, and now I’m a super-hero coding monster. Also, my breath cures cancer. Hooray for Command-T!”

I’m tired of posts like this too.

I want to make something clear: there are really good reasons that you can’t write a commit script, track your performance and expect it to quadruple. There are good reasons not to even try:

Reason #1: I was intrinsically motivated; that’s why it worked

In psychology and the study of education it’s well-known that intrinsic motivation is everything in learning and self-improvement. I already yearned to improve my own effectiveness, that’s why I started doing this. It fascinates me, it calls to me. That’s why it worked.

If you already long deeply to improve yourself, you will do with or without my advice. If you don’t, then drawing little graphs won’t help either. There’s no reason to believe that blindly going through the same motions I did will produce the same result for anyone else.

Your motivations can and do change over time though and maybe reading this will help to motivate you too - I hope so!

Reason #2: never replace intrinsic with extrinsic motivation

It turns out that offering employees bonuses or performance-related pay can be harmful. In some cases - such as programming - an external motivation, such as a bonus, produces measurably worse performance than an internal motivation, such as the love of coding.

Worse still, such an extrinsic motivation will often replace an intrinsic one, damaging performance and job satisfaction.

If you start measuring your performance, you may end up replacing your intrinsic motivation to do a great job with an extrinsic one to make the numbers go higher.

If you start measuring other people’s performance, you certainly will.

Be very, very careful.

Reason #3: these stats don’t measure programming excellence

So, should I ask for a 400% pay rise now? Am I four times as valuable to the company than I was last year?

No and no. I like to think I’m somewhat more valuable to the company now, but not by a factor of two, let alone four! You see. these stats don’t encompass everything a programmer has to do to produce awesome software; they don’t actually try to - they have another purpose.

It’s like running and football: professional players care about how fast they can run 50m. They even practice sprinting during training. It has some value to them. Yet if you ranked all professional football players by how quickly they can run 50m, you wouldn’t find all the top players at the top.

How would a football team composed entirely of olympic sprinters fare? Exactly. Running fast is one tiny part of playing football. Resolving bugs efficiently is one tiny part of writing excellent software.

Stats such as these are useful for your own training, for measuring and improving one specific aspect of your competence but they are worthless other purposes. Don’t use them for:

1. Comparing developers to each other
2. Measuring or motivating a team
3. Hiring new people

It should be clear that single-figure statistics are useless for these goals, but people love graphs and they love putting numbers on things so I’ll state this as clearly as I can:

It doesn’t matter how many bugs you fix per month if your code sucks and the UI looks like a Fukushima control panel after the explosion.

The bottom line is that you can use these stats to measure your own performance increases, but once your goal becomes to increase the stats, you’ve already lost.

Good luck!

Postscript: If you’re interested in seeing my sorry collection of hacked-together python and bash scripts, leave a comment to that effect and I’ll put together a “Metagame: Nasty Hack-Filled Scripts” post for you with all the sources. I’m happy to talk about this on twitter, too.

Edit: Here’s a link to the HN discussion and the Reddit thread for this post.

I’ve been bored at work for many reasons at many different times, but three things stand out as real killers:

1. working on the same project with the same people[1] for years and years,
2. using the same old languages and tools (statically-typed: yuk!) instead of the new hotness,
3. being forced to work on maintenance instead of new features, or on small parts of an existing product instead of creating something new.

These are the symptoms of a problem, not the cause, and I think most jobs will have elements of them. But surprisingly it turns out that - for programmers at least - boredom is a choice. Recently, I chose not to be bored. I chose to think one abstraction level higher. I chose to play the metagame.

Me: a case study

I started thinking about programmer performance a while ago. Everybody will tell you that you can’t measure programmer productivity, but this is at best a half-truth. We can, and we should. Perhaps what we shouldn’t do is use those measurements to compare programmers to each other, but we can definitely measure ourselves.

When I started trying to measure my own productivity, it g

ot me thinking a lot more about:

1. my workflow - I check email, pick a tracker ticket and work on it.
2. the most repetitive parts of my work - why does it take 10 clicks and 3 windows to commit a change?
3. the core of my job - I’m paid to solve non-trivial problems, not to sort email and adjust ticket priorities.

It really doesn’t matter what anyone’s day job is; whether you’re working on sporting result clips for Google mobile or maintaining a 20-year old internal accountancy package the metagame is the same. The product may be pointless but the metagame matters to us all. It’s about:

1. surrounding ourselves with tools and systems that make us more than human, a cyborg-like super-programmer - who doesn’t want that?
2. automating away the boring parts of work, so that we’re left with the interesting, non-trivial decisions and problems.

When I started I was working on bugfixing before a major release, which is very amenable to this sort of thing because there are lots of small, well-defined packets of work. We were about 9 weeks from release and my typical day looked like this:

1. Check and respond to my emails
2. Check the test machines and fix any system or build failures
3. Take a high-priority bug from our tracking system
4. Try to reproduce it locally
5. Track down the cause and write a fix
6. Commit the fix, update the ChangeLog, resolve the ticket
7. Repeat from step 3

It was quite easy to measure my own performance in terms of the number of bugs resolved per week. Sure, it’s not a perfect measurement - I could also have factored in the priority of the bugs and so on, but getting an accurate figure wasn’t nearly as important as getting some measure, however rough.

With the goal of increasing my rate of bug fixes motivating me, I started laying into my workflow and adding automation wherever I found an opportunity to speed things up:

1. Don’t waste clicks navigating email

Email checking was clearly not related to my core metric (bugs fixed) but I couldn’t just ignore emails, however much I wanted like to. Instead, I added a bunch of extra filters to brutally cull all the emails I didn’t really need to read, then found out how to get keyboard shortcuts working in Gmail (hint: add &kbd=1 to the URL). Now I was whipping through the emails by pressing [ to read-and-archive for 95% of them, occasionally opening extra tabs to remind me of various actions I needed to take later in the day.

2. Don’t type ChangeLog entries by hand

Next up was the ChangeLog. I don’t know if you keep a ChangeLog at work, but we have done for a while. The idea is that with each commit you log the broad reason for the change, then write a one-liner for each function you’ve changed, along with the filename, class and method. Writing these by hand was incredibly tedious. Rather than dump it altogether (for it had its uses) I invested a few hours in writing a 147-line python script that generated the entry stubs automatically with inline diffs for each function changed. I added a handful of vim bindings to jump between the ChangeLog entries and replace each diff with a one-line description of it’s purpose. Suddenly, writing ChangeLogs was ten times faster even somewhat fun.

3. Yes, I said Vim

It’s no coincidence that I started using MacVim shortly before my metagame trip. Vim is awesome for exactly one reason:

1. A vim script is exactly the same as the keypresses you use when editing

Writing a script in vim is a simply matter of typing the same characters you normally would to achieve that effect (which is why using hjkl for movement isn’t as stupid as you thought it was). I cannot imagine a lower barrier to customizing your editor. The end result is that vim makes it trivial to optimize all sorts of little common operations; it’s the first editor whose macro functionality I’ve used on a daily basis. It’ll change your life. Learn it. Love it. It’s worth it.

4. Make reproducing bugs trivial

My previous attempt to make my job more interesting had been to volunteer to rewrite our clunky, frustrating GUI-click based test system with a new, streamlined one. Since we use Qt, I added a QScriptEngine which essentially lets you write a small C++ API for your application and then call that from javascript files.

This was a lot faster and less error-prone than manually baby-sitting fragile GUI-based testing as we’d done before, but the real benefit was that I had complete control over the system. Every night it ran tests across dozens of platforms and reported any new failures directly to our tracking system. I started adding extra code to our test-runner (a 1,500 line python script) to make reproducing and fixing bugs stupidly easy. Each test report now includes:

1. the stack trace and variables for any segfaults
2. the core files for any segfaults in external programs during the run
3. all log files, stdout and stderr
4. screenshots of the GUI should a test timeout
5. a one-line, copy-and-paste command to re-run the test

Adding this one piece at a time wasn’t a lot of work, but it has saved us hours and hours and hours.

5. Add ssh and bash shortcuts

I spend a lot of time on our test machines, so I added hostname auto-complete to my bash shell, added single-line aliases to change to the test directory, check out an overnight build, view the build and test logs, change to the test user and so on. I learned to love .ssh/config, and so should you.

Saving a few keypresses doesn’t sound like a big deal, but it makes me happy every day when I type:

cs
smoketest -kri 14956

to switch to the latest test directory, update its license file and re-run the bug reported in ticket 14956. I could write an essay just on this:

every single element of cognitive burden removed makes it easier to stay focused on the task in hand

Not having to remember a hostname, or a compiler directory, or keep track of which system I’m on, streamlining all that away makes it trivial to keep thinking about the problem and not the distractions.

6. You can’t have too much Vim

I ended up adding lots of vim shortcuts specific to our systems. It’ll look up and fill in the title of any bug ticket for me. It’ll commit my current changes, automatically taking the ChangeLog entry I just wrote as the commit message and optionally resolve the bug ticket associated with that change using the same message. In one command.

Typing that one feels really good.

Bundling up all the points at which I’d otherwise have to switch to a browser, wait for a page to load, this was accidental but brilliant. Waiting is the death of focus. By wrapping it all together into one fire-and-forget command, I can just say the word and then start choosing the next bug to fix, keeping my momentum and focus intact.

I’m not done

Every week I find new places to tweak my setup or semi-automate more tasks. I’ve started producing beautifully-formatted HTML change diffs for code review on my iPhone, while reclining in one of Combinat56’s Sumo bags with a cup of tea. I’ve added extra notifications when a build fails or a test machine runs out of disk space.

I will never be done, because I can always keep moving one abstraction higher until I’m purely working on the most interesting problem of all: how much of a programmer’s job can be automated?

A nice side-benefit is that much of the code I write to automate things is not production code; I can write it however I like, using whichever languages and tools take my fancy. Yet it all goes towards making me - one day - into an extraordinary programmer.

Playing the metagame: advice

After working like this for six months or so, the following pieces of advice seem like the most important to remember:

1. Don’t be afraid. Spend time improving your efficiency instead of hacking away with a blunt axe. There can be a lot of pressure “just to finish this first”. Resist it. Take just one hour and use it to write a script to help you in some simple way, or improve your bash aliases, or your ssh config, or your mail filters. Once you’ve seen how much impact it makes, you’ll feel a lot better about doing it again.
2. Semi-automation is better than full-automation. It’s easy to get carried away and try to write your own email client, or use bayesian filtering to answer all your emails. Don’t bother. Write the simplest, smallest thing you can to speed you up and then get back to work. If it helps, you’ll find yourself building on it naturally over time. It’s better to have one-key read-and-archive than a custom email client for your iPhone.
3. Keep scripts specific. Don’t try to over-generalize them and solve everybody’s problem; that’s not going to be practical on the side. Make your scripts specific to you and your workflow. If they’re great, you can extend them to the team. If they love it, you can extend it to the world and put it on GitHub. But do it in that order.
4. Use vim. It feels weird at first. It takes a couple of days to get used to. Set it up with nice plugins and themes right away; after a couple of weeks you’ll never look back.
5. Optimize the things you hate, not the things you love. The goal is to build up scripts, commands and systems that feed you a flow of meaningful decisions to make and interesting problems to solve, while automating away the dross. Resist the temptation to start by optimizing programming or code-writing; attack all the distractions first.
6. Never let yourself wait for something. If you catch yourself typing a command then waiting for it to finish before you can type the next one, then that’s an ideal place for a script. Waiting will kill you; make the computer wait for you. For some tasks, like compiling, write scripts that’ll automatically launch the program when compiling finishes, perhaps even re-running the test case if possible. Make your computer notify you when it’s ready, audibly or by throwing something up on the screen. In the meantime, you can be reviewing code or looking for a second bug to start work on, or taking a walk with a cup of tea. Anything but HN and Reddit!
7. Write in all the fun languages you can’t use at work. I can’t recommend python enough for this sort of thing, as it’s huge supply of libraries combined with pleasant syntax make life a breeze. I guess Perl would be great too. But get used calling external programs and input / output streams and you can write in Node.js or whatever interests you most this week.
8. Measure your performance. Write a simple script to measure the number of commits you’ve made. Or the number of lines you’ve changed. Or the number of bugs you’ve fixed. Or the number of customers you’ve helped. Just pick something that’s trivial to measure even if it’s far from perfect, and start. Add more things whenever you’re afraid you’re over-optimizing or gaming the measurements you’ve got so far. Seeing your own performance climb gives a real sense of satisfaction that might be otherwise missing in your daily work. Even if you don’t enjoy or value the internal actuarial calculator you’re writing, you should care deeply about making yourself a better programmer - this will stay with you your entire career.

That’s it - print this list out, stick it next to your monitor then close the browser and start playing the metagame! You weren’t going to do anything more productive in the next hour anyway, right? ;-)

Update: just in case you’re not heading off to work, the HN thread is here and the Reddit thread is here.

[1] Disclaimer: I work with absolutely first rate programmers, people I learn from every day and yet there’s still an attraction to meeting new people too; I shudder to think what it must be like to work with dull people for years and years…