Yield Thought

it's not as hard as you think
formerly coderoom.wordpress.com

Writing AI to play games is a special kind of crack for me. Seeing my AI face off against other people - or better yet, AIs written by other people! It’s so… I don’t know how to describe it. I cannot resist its siren call.

I once lost several months of my life writing python-based AI for an open source game called The Battle for Wesnoth. What started as a fun way to learn python quickly turned into a several month-long obsession of unhealthy proportions. Towards the end of it I had an AI with an ego-inflating 98% win rate versus the default AI across all in-game factions and a selection of the smaller maps.

Then, on the very edge of victory, it all went horribly wrong.

Pride comes before a fall

My downfall begins on a hot summer’s night in Munich. I can’t sleep and I’m thinking about Wesnoth. I long to capture the last 2% and have an AI to share that wins every match, but I’ve exhausted all of the small tweaks and improvements I can think of.

Tonight I decide the time has come for a significant rewrite. It will put the AI on a much more strategically sound basis and practically guarantee total domination! I refil my glass, sit down and get started.

It take several evenings to finish tidying the changes up, but finally the moment for a full run-through comes. I hit the button and send my newly revamped AI into a gauntlet of battles against its soon-to-be-vanquished opponents. But something is wrong. It’s losing a match! Now another! And another! After thirty minutes or so the final figure comes back: 72%. SEVENTY. TWO. PERCENT.

The road from 72% to 98% took me almost a month of occasional evenings the first time around. What do I do now? Revert the changes and try something else? Or keep working on them, possibly throwing my time away for nothing?

Questions whirl around in my sleep-deprived mind, taunting with their unanswerability. Are these changes fundamentally worse than the previous strategy? Or are subtle bugs causing my AI to make stupid mistakes? If I spend two more weeks fine tuning this version, will it ever overtake my previous plateau or am I wasting my time?

In one especially lucid moment I feel a sudden connection to the hopeless fate of the hill-climbing algorithm, never aware whether it is stuck in a local minimum or not, always caught between tenacious hope and the terrible, crushing despair of futility.

Charlie Brown says that the tears of adversity water the soul. I learned that a single number just isn’t enough feedback to meaningfully understand the performance of a complex system. I also developed a deep sympathy for anyone who spends their life A/B testing website changes.

A few weeks later the Wesnoth team discontinue the python AI interface and I am rehabilitated back into normal life, but the experience makes a lasting an impression on me. How can we write better AI? How can we test and understand the behaviour of code that interacts with an overwhelmingly complex, stochastic environment in unpredictable ways?

Back on the wagon

The seasons change and years pass peacefully by. Purely by chance I stumble upon the Wesnoth website again and notice they have quite a decent lua-based AI subsystem in place now.

Unable to resist, I clone and compile the latest version and start learning how to use the new lua interfaces. It’s not long before I have a skeletal AI - in every sense of the word - managing a 30% win rate with lots of low-hanging improvements still unexplored. Yet I am uneasy. I can feel the same problem out there, an undefeated nemesis lying in wait.

I live by the sea now, so one evening I take my notepad and pen down to the beach, write down the problem as clearly as I can and stare helplessly at the shimmering waves, waiting for inspiration.

Despite vicious sea wind’s tenacious attempts to strip the flesh from my bones I manage to sketch out an idea for a workflow that feels like it has potential. As it turns out - to my own surprise and joy - it’s absolutely brilliant. Sometimes the Feynmann algorithm is the only one you need.

Machine learning, or is it machine teaching?

The problem I wrote down is that I’m drowning in an ocean of raw data without an easy way to understand the stories within it. Each automated match consists of hundreds of moves from a wide range of different units and to get any meaningful comparison I really need to run dozens or even hundreds of matches per change. Abstracting all of that as a single percentage throws too much information away but watching fifty replays each time I tweak a number is never going to happen.

At the beach I realized I need a machine to watch them for me.

This is how I built one.

Step one: visualize a single battle

Realizing I needed something between clicking through an entire replay and a single win/loss figure I started looking through battle logs picking out other statistics that seemed strategically relevant.

In Wesnoth there are 5 key metrics that together give you a pretty good feeling for which way the battle is going. I logged just these over time for each battle and added some unicode sparklines for readability:

yt_simple vs ai_default_rca  |  winner: 2 
units: ▄▄▄▄▄▄▄▄▃▃▃▂▂▂ 49 53 49 45 49 49 49 46 42 38 30 21 19 18 
cost : ▃▄▄▄▄▄▄▃▃▃▂▂▂▂ 40 48 45 43 44 44 44 41 38 31 27 21 22 18 
gold : ▄▁▄▄▄▄▄▅▅▃▃▄▃▃ 49  0 57 54 52 46 54 60 69 40 38 53 41 40 
vills: ▁▁▄▄▄▅▅▅▅▄▄▃▃▃  0  0 55 52 52 63 61 59 61 55 49 30 33 33 
inc  : ▃▂▄▄▄▅▅▅▅▄▄▃▃▃ 33 24 54 52 52 61 60 58 60 55 49 33 34 34 

Even so, reading through fifty of these reports at a time isn’t very illuminating, even when they are sorted. Sorting is my favourite cheap visualization trick but here I needed something more.

Step two: the unreasonable effectiveness of normalized compression distances

Over the past year I’ve been fascinated by the usefulness of the normalized compression distance (NCD) as a similarity measure. Basically it states that the degree of similarity between two objects can be approximated by the degree to which you can better compress them by concatenating them into one object rather than compressing them individually.

You can use this to detect the similarity between music, images, DNA and all sorts of arbitrary things wth surprisingly good performance. It’s one of the coolest and most implausibly effective hacks I’ve come across and I thouroughly recommend reading Vitanyi, Cilibrasi and Cohen’s papers on the subject.

NCD was perfect for this problem. I built up a similarity matrix by compressing my visual output summaries then used that to cluster them into an unrooted binary tree. A small python script injects some extra details to dot’s SVG output and suddenly I am looking at this:

The nice thing about having these in SVG files is they are mildly interactive - moving the mouse over any particular node shows the run summary in the bottom-left, as you can see for run 11 in the image above. The NCD clustering does a really great job here – the clusters really do seem intuitively similar and semantically meaningful even after deeper investigation.

Of particular interest are the battles that were lost but are clustered close to several wins. These scream “everything was fine until something unexpected happened” - a bug or unforseen situation the AI has blindly wandered into again. Watching the replay for one of these battles almost always shows a (sometimes rarely-occurring) bug at work. Just fixing these improved my AI massively - machine assisted debugging at its finest!

Step three: profit! My new machine-learning boosted workflow

When I sit down to improve my AI it now looks like this:

  1. Code up whichever improvement or brilliant/stupid idea I’ve been dying to try out
  2. Hit a key to run 50 battles in parallel using 8 cores, cluster the results and produce a prettified SVG file
  3. Get a feel for the overall shape of the results. Are there lots of quick wins, or are matches more often drawn out? Are there interesting clsuters of wins and losses?
  4. Tap on a few nodes to understand what is represented then pick one to look at in more detail, e.g. a loss surrounded by wins, or an example from the center of an interesting cluster of losses.
  5. Type in the number of that node into my terminal to launch Wesnoth and view the replay of the run from that node. I can click forwards and backwards through the match, watching for mistakes both obvious and subtle in the behaviour of my AI. Sometimes I look at a couple of runs for comparison, such as a similar win/loss or two similar lost runs.
  6. Sometimes I can see what the AI did wrong but don’t understand why. In this case I can pull up the AI debug log for that numbered run, jump to the turn I’m looking at in the replay viewer and read in depressing detail the reason my thief decided to go toe-to-toe with a mounted Lancer in waist-deep water instead of e.g. nipping onto that nice safe village and stabbing him in the back for 2x damage.
  7. Come up with a new idea to improve the existing code and repeat from step 1.

Working like this is so beautiful because there’s a smooth multi-level mapping from the complex emergent behaviour in the game all the way back to the actual lines of code that evoked it and it’s so frictionless to move between levels of detail that it’s joyful in and of itself! It’s also extremely effective.

Some Promising Results

Within a week my lua AI has reached a 98% win rate and I haven’t even brought out my best tricks yet.

Every time a change decreases the win rate I’m able to quickly find the group of replays demonstrating the negative impact and follow through from there to the debug logs to the code, where I can fix it. It’s a whole new world compared to the way I worked last time, with just a single figure telling me “better” or “worse”.

I would be surprised if writing AI is the only application for a workflow like this.

A/B testing might benefit from a similar setup – take the logged or recorded traces of each visitor’s interactions with the site and cluster them so that when a test decreases signups you can look at the clustering and see which behavioural subgroup is failing to sign up, then go and watch a few sample recordings to understand why they might have changed their behaviour.

I don’t write a website analytics package any more, but if you do and you’d like to experiment with this I’d be interested in working with you. This stuff is a lot of fun!

There must be a lot of other situations that would benefit, too. Do you know of any others? Let me know, I love learning new things!

Update: I’ve added a few more links and technical details to the Hacker News discussion.

I kept wanting to check this on my iPad so went ahead and hacked together a cron job and static Google AppEngine site for it. It’s the daily probability that any given team will win the world cup, visualized over time.


I’ve been having fun looking at the betfair market odds of a team winning the world cup over time. The interesting thing isn’t the wins and losses, although those are very clear, but how they affect the other teams chances.
The vertical axis is the % chance of winning, as predicted by the betfair market after the end of play each day.
In some cases a team wins but their performance was so unconvincing that their chances of winning the title barely increase. Relative to the points they received, they might actually decrease - witness Brazil’s early matches.
Germany goes from strength to strength, the only team that’s consistently impressed so far.
It’s also surprising how few teams are considered to have any worthwhile chance by the market - and these are getting fewer all the time. Judging by the size of the increase seen when the Netherlands thrashed the reigning world champion, the market’s really really convinced that one of the favourites will win.
Of course, I’m hoping England will have found the perfect way to build Rooney into their forward four and will put in a blazing performance tonight. Come on England!
Whether the market is rational and efficient is also interesting, especially if you like writing trading bots. Some of the movements suggest it might not be, but I really need to annotate the graph with match results to make them obvious.
I hacked this together a few nights ago using Python 2.7 (oh, python 3, I’ll start using you one day), Selenium+ PhantomJS (who has time to sign up for APIs?), Chart.js, Chart.js.legend, Linode, Ubuntu, GNU screen, Chrome (I like the profiling chart) and my beloved Vim.
Truly we stand on the shoulders of giants!
Anyone interested in seeing this online?
P.S. If you’d like to rewrite the graphing using something more interactive (d3.js would be my choice) then drop me a line, it’d be fun to collaborate!

I’ve been having fun looking at the betfair market odds of a team winning the world cup over time. The interesting thing isn’t the wins and losses, although those are very clear, but how they affect the other teams chances.

The vertical axis is the % chance of winning, as predicted by the betfair market after the end of play each day.

In some cases a team wins but their performance was so unconvincing that their chances of winning the title barely increase. Relative to the points they received, they might actually decrease - witness Brazil’s early matches.

Germany goes from strength to strength, the only team that’s consistently impressed so far.

It’s also surprising how few teams are considered to have any worthwhile chance by the market - and these are getting fewer all the time. Judging by the size of the increase seen when the Netherlands thrashed the reigning world champion, the market’s really really convinced that one of the favourites will win.

Of course, I’m hoping England will have found the perfect way to build Rooney into their forward four and will put in a blazing performance tonight. Come on England!

Whether the market is rational and efficient is also interesting, especially if you like writing trading bots. Some of the movements suggest it might not be, but I really need to annotate the graph with match results to make them obvious.

I hacked this together a few nights ago using Python 2.7 (oh, python 3, I’ll start using you one day), Selenium+ PhantomJS (who has time to sign up for APIs?), Chart.js, Chart.js.legend, Linode, Ubuntu, GNU screen, Chrome (I like the profiling chart) and my beloved Vim.

Truly we stand on the shoulders of giants!

Anyone interested in seeing this online?

P.S. If you’d like to rewrite the graphing using something more interactive (d3.js would be my choice) then drop me a line, it’d be fun to collaborate!


The NSA “is gathering nearly 5 billion records a day on the whereabouts of cellphones around the world.” (Washington Post)
Join us in protesting the National Security Agency’s wide-ranging invasion of privacy.
Take action →

This is not really just about the NSA; our whole system of government and national competition is set up to make pressure towards mass surveillance irresistible to many parties.
What only national-level budgets can do today, corporations and extremist groups will be able to do tomorrow. Or next year.
That’s why this is the right time to stand up for privacy.


The NSA “is gathering nearly 5 billion records a day on the whereabouts of cellphones around the world.” (Washington Post)

Join us in protesting the National Security Agency’s wide-ranging invasion of privacy.

Take action →

This is not really just about the NSA; our whole system of government and national competition is set up to make pressure towards mass surveillance irresistible to many parties.

What only national-level budgets can do today, corporations and extremist groups will be able to do tomorrow. Or next year.

That’s why this is the right time to stand up for privacy.

How has your experience been relating to the use of a bluetooth keyboard with the iPad; specifically with the way iOS requires the activation of Voiceover to increase remote keyboard functionality. Personally I find the accessibility features in iOS second to non both keyboard and voice but wanted to pole for your experience and findings as I and I'm sure others may be interested to learn such things. Frustratingly how to copy and paste code outside of anything but screen or TMUX. Thanks (:

Hi Edg3e,

I hadn’t looked into using Voiceover to activate app switching and so on - that’s… interesting. It takes away one of the few things I preferred about the Surface, whilst adding some of its own new annoyances. Progress, I guess?

I sympathize with the copy and paste - often I wanted to take some text from the console and put it into a bug ticket; I ended up writing a small script that uses our bug tracker’s API to add a comment containing text from the command-line, and then extending it to automatically include the last stack dump and log file output. Surprisingly, this made me more productive than I had been with manually copying and pasting things all the time. That’s something I found a lot while working with the iPad/Linode combination - things that were too uncomfortable to do the ‘normal’ way ended up being shoehorned into a ‘console’ way which often resulted in quite a bit of time-saving - and, crucially, attention-saving.

Typing a short command at the console gives me fewer temptations to check my email or notice a support ticket has been updated than switching to the browser does.



"I never thought I’d write a blog post in Microsoft Word, but here I am doing just that," begins my first blog post on the Surface. Written lying on a rabbit-shaped bench overlooking the sea, with a gentle breeze cooling the heat of the sun on my skin, all is well. A week later when I come to edit and upload it, I discover that Word has lost it[1].

Act One: All good things come to an end

This story begins back in January. I’d been curious about trying a Surface + Linode combination instead of my 2 year-old iPad + Linode setup. The allure of a full-fat browser that can run Gmail and iPython Notebooks, combined with a keyboard-centric OS has long tempted me. I hadn’t switched, though. Somehow, I had been waiting. At the end of that fateful January amidst the snow and the cold, everything changed.

I was promoted into a management role.

Overnight I find my entire interface to the rest of the company changes. What used to be IM and Mercurial is now emails with attached Word and Powerpoint documents. Within a fortnight the iPad + Linode setup collapses; neither iOS nor Linux nor Google Docs offer credible alternatives for reviewing, editing and returning Office documents and I grit my teeth, install Windows 8 and Office on my laptop and get to work.

Fast forward to April, and I’m wandering through a cherry-blossom filled Washington, exasperated by two months of being tied to a laptop, with all the power, size, screen, weight and fragility restrictions that go along with one. I’ve mostly stopped working in cafes and outdoors, but I miss it. As I walk I weigh up the alternatives. A 13” Air would be a massive step back towards the freedoms I’ve grown used to, but the fanless, robust silence of a tablet continues to attract me, and I don’t want to go back to managing local storage any more than I can help.


Reaching a decision, I drive to a cathedral-like mall and walk out again a few minutes later with a Surface RT and type cover. The time to try a Surface has come. If anything can let me work as flexibly and freely as my iPad in this new corporate world, the Surface can.

Act Two: Flashback

In the following glorious three months of summer and sunshine at my new home by the sea, I get to know the Surface, its glory and its shame. It works, I use it every day. Yet something is missing, something isn’t right. I just can’t fall in love with the Surface.


Back in primary school, my class made Christmas cards for our parents one year. My artistically-gifted friend, who knew the difference between a B and HB pencil, drew out a simple, bold cracker design with a simple twist - you can pull apart the cracker to see the message. No more, no less, but somehow he managed to spend the whole hour working on it.

I thought the pop-up idea was great. My card had a cracker too, and when you opened it there was a fireplace and tree inside and there were extra tabs to pull to make the fire go out and Father Christmas come down and you could open one of the presents under the tree to see my name and I’d fitted most of the words in without squashing them and when I showed the resultant sticky, thumb-printed mess to our teacher he asked me what on earth that was and told me to throw it away and make another.

I didn’t make another, but I did take a long, hard look at my friend’s card. Mine did more; why wasn’t it better? In that moment, I first learned about the relationship between elegance, simplicity and beauty, and my card with all its features and messy edges had none of these things.


Using the Surface gives me the same feeling. I can do more with the Surface, but it is not beautiful, nor do I enjoy using it - or being seen using it. It’s so obviously a business device that putting it on the table in a cafe feels… awkward. Like an imposition, doing office work in a leisure place.

The experience is often disjinting. Rotating the device results in a weird “pop” animation that often leaves desktop windows or even the start screen itself distorted and out of place. It persists in referring to itself as my “PC”. The keyboard doesn’t auto-appear when text fields have the focus in desktop applications. Metro IE sometimes decides to reload all the old open tabs when clicking a link in a mail switches to it, stealing the focus from the new page I just tried to visit. The kickstand is at just the wrong angle for everything and the keyboard, while fine on a table, is spongy and occasionally misses keypresses when used on a lap. Word doesn’t save your work unless you click on a 3.5” disk icon regularly. There are a hundred small things that make the whole experience feel fragile, confused and unsatisfying.

No, I cannot love the Surface. But neither do I discard it. Despite its failings, it does what it claims. It works, and it lets me work, flying through my Gmail with keyboard shortcuts, effortlessly opening, reviewing, changing and reattaching Office documents to emails. Sometimes, it even feels incredibly cool - using an iPython Notebook running on my Linode to graph and explore data from the tablet is absolutely great, even if it’s only possible thanks to the single-click jailbreak.

Act Three: Surprise

You see, there’s something else, something that changes things in ways I never expected: I can read the Surface’s screen in direct sunlight.


While I can just about work on my iPad in the shade of a tree, the Surface screen is clear and bright lying in direct sunlight in the hammock on my balcony, on a pier at the beach, outdoors in a café or on a rug in the garden. This changes the game again. Working in a natural environment can be both beautiful and liberating. It’s just a shame it’s coupled to a device that’s otherwise somehow disappointing.

Visiting my friends in Munich is typical of my mixed feelings for the situation. It’s a 7 hour train journey and a 7 day trip. I take the Surface for working on but also pack the iPad for playing games, so now everything weighs as much as a small laptop would anyway. My friends in Munich consistently tell me how healthy I’m looking. I put it down to my suntan from all those mornings in the hammock with the Surface.

I prefer the experience of working on the iPad. The amazing, unexpected mental freedom of having all my state on an automatically backed-up, always-on Linode coupled to a simple, essentially stateless window onto that world that I could toss around like a paperback book. The ability to open and close it at will, let the battery run flat, leave apps open with the contents unfinished and never give them a second thought.

The Surface doesn’t persist application state. Every week it threatens to restart itself within 2 days to install updates. If I haven’t manually saved work when it does that, the work is lost. Some things are saved on SkyDrive, some on the local disk, which isn’t automatically backed up either. So now I am forced to be aware of this leaky abstraction, to manage the remote and local state and at this point I might as well just have a MacBook Air and be done with it.

It lacks a zen-like calm.

And that’s what this comes down to. If you’re mostly doing non-web development, go for an iPad+Linode. If that combination can’t do the work you’re doing - like the MS Office editing I need to do now - then it’s a choice between the Surface and a laptop, not an iPad. Do I recommend the Surface for working like this? I don’t know. It’s been months and I still don’t know.

I would miss working outside if I had to go back to my laptop full time. Cycling into nature and working in a beautiful spot is fabulous. But these are things the screen lets me do, not the things the Surface lets me do. I expect to see similar screens on other devices soon. A good screen is not fundamental to the form factor or the manufacturer. There are probably options out there already that I haven’t seen because I never thought to look before.


I do know that I almost never use the Surface as a tablet. I use it like a little laptop. Maybe I’ll find the perfect combination of powershell scripts and metro apps to recreate an elegant workflow, but I’m not really motivated to because while I could use shell scripts and python programs on the Linode to work around the iPad’s email limitations I can’t do anything work around the Surface’s deeper usability flaws.

Every time I use it I miss my MacBook a little more.

Update: HN discussion here (including my thoughts on Microsoft Phone).

Update: I forgot to mention my Working in the Cloud mailing list. Articles like this appear there first, sign up if you’re interested in how things go from here!

[1] Actually it autosaved a file, and when I next tried to open an attachment during a conference call I ignored the annoying little restore blah blah tab. I’ve since learned not to do that. But I shouldn’t have to.

Note: this is (most of) the content from the first post to my Working in the Cloud email list - more are prepared and will go out soon, so if you don’t want to miss the rest you can still join the list here.

About a week after I published the iPad + Linode: One Year Later post, my employer decided to buy me a shiny new laptop for CUDA (GPU) development - a Dell XPS 15 with 8GB RAM, a quad-core i7 CPU and a 512GB SSD drive. It was shiny!

And it was fast. Compile times dropped from 53 seconds to just 20. Test suites ran faster. With 8 GB memory I could run larger tests, or more concurrently. Disk space - always a limitation on the Linode 512 - was no longer a problem.

I’ll be honest: after the first day, I began to have doubts about my future in the cloud.

There were one or two cracks in the armor; a laptop keyboard just isn’t as nice as the Apple Wireless one. It gets unpleasantly warm and it’s just a bit too high off the table. I tried an external keyboard, but then I’m sitting too far away from the screen.

I also missed touch. I never expected to say this, but moving the pointer around with a touchpad or even an external mouse just feels slightly awkward now. It’s less immediate; one more thing for my subconscious to work on.

Having a desktop-class browser is really, really nice though. GMail is more responsive and I can use Google docs like a Real Person again. I can even attach things to emails! Hallelujah!

The one thing I don’t really notice while programming is the screen. Everybody asks me how I can work on such a small or low-res screen with the iPad. Really, I don’t notice the difference unless I’m doing a 3-way merge and need the extra horizontal space.

I spent a whole week working on the new laptop. It was a busy time at work and I was multitasking like an over-caffeinated rabbit, flicking back to my emails every few minutes to clear the next dozen that had arrived since I last looked. Meanwhile, my iPad lay in the corner quietly discharging its battery into the void. 

Until Thursday afternoon. It was a splendid autumnal day and as I stepped outside at lunch to pick up some fresh bread from the bakers something in the air called to my soul. You can’t sit inside on the last fine day of autumn, it whispered to me, come with me and be free.

I went back inside, packed up my iPad, some water and a rug into my rucksack and set of, into the great outdoors. Twenty minutes later I was lying in the dappled shade in the vastness of Munich’s English Garden, feeling the breeze playing in my hair as set up my 3G tethering. This is right, I thought. This is the life! And yet at the back of my mind was a guilty question: am I wasting time out here? Would I be more effective sitting back at home chained to The Machine?

I logged in and carried on hacking away at the same bug I’d tried and failed to track down the day before; a subtle problem that spanned several concurrently-running services in our system. I’d grown used to the rapid build/retest cycle of the new laptop and was finding the slower pace of the Linode increasingly frustrating. The air was beginning to turn chill and suddenly this wasn’t where I wanted to be or what I wanted to be doing.

I queued up another build and a set of offline tests to run and checked the time; it’d take a good five to ten minutes before the results were in. Time to move. I told the system to ping my phone when it was done then packed up and got back onto my bike, heading South to find a nice warm cafe to finish the afternoon in.

Back on my bike, gently coasting along the tree-lined paths, my muscles warmed up and my mind relaxed again. My thoughts turned back to the problem; not wanting to waste any more time I wondered how I could most effectively narrow down the scope of the bug. As I was crossing a small bridge I felt the buzz of the notification in my pocket; the builds were ready. Nearby was a small sunlit glade; I settled myself down between a small tree and the stream.

Before logging in on the iPad, I lay back, watched the leaves fluttering against the darkening blue of the sky and finished thinking about the tests I’d run; what the various outcomes might actually mean, and what my next step could be. Once I knew exactly what I wanted to try next, I opened the iPad, logged in and did exactly that - no more, no less. Thirty minutes later I’d solved the problem.

Riding back home, I realized I’d spent the entire afternoon thinking about and working on the problem. Not checking my emails, not writing a quick reply in some barely-related feature discussion. Not reading Hacker News. Just focused on the most important problem of the day.

The same thing happened a week later when I had to write a press release. I’d spent all week putting it off, finding ‘more urgent’ things to do in my emails, attacking small development problems. By Friday afternoon I’d run out of time. I closed the laptop and took my iPad to the window ledge, where I sat with a cup of tea and looked out across the grass, letting my thoughts come to rest.

Then I started typing. Typing on the touch screen is slower than a keyboard, just as compiling on a Linode is slower than on a quad-core i7. Yet again, here the bottleneck wasn’t my typing speed, but my thoughts. The slower pace became meditative, the lack of distractions absolute. An hour later I was done. Later that night our CTO replied saying that he loved it, calling it “quirky, refreshing, and extremely likely to get the coverage we need.”

I’m coming to the conclusion that the speed and generality of a laptop or desktop system is optimizing the wrong thing - it’s increasing the rate at which my brain receives distractions from the true bottleneck in my work: measured thought, repeatable inspiration.

Yet I have one more test left to try, and one that I think might interest you: what if I try to get the best out of both worlds and use the iPad to remotely connect to my work laptop? This is something you could try, too - after a little bit of setup you could take your tablet down to the building cafeteria, or library, or outside, or home, and get a taste for what working on it is like without having to sign up and set up and entire remote server.

I’ve now set this up myself using an iPad and Ubuntu 12.04 laptop and will send detailed instructions on how to do it in the next email. If you want to receive it, click here to join the list.

Thanks, and until then, happy cloud working!

A year ago I said goodbye to my trusty MacBook Pro and started working exclusively on an iPad + Linode 512. It was an experiment at first - one that I never thought would last.

Twelve months later and I find I’m still working like this. A combination of Vim and GNU Screen for development, Pages for writing, Keynote for presentations, Jump and VNC for unavoidable X windows work, Mobile Safari for web apps and a hefty dose of python scripts to smooth off all the edges. I use it for development, for presentations, for my side projects, for everything. 

I really mean everything, by the way. A high-speed baby bottle demolished my MacBook Pro screen months ago and I still haven’t got around to getting it fixed yet.

I love this setup, but it isn’t perfect. I’ll be making some changes for in the coming months. Before I get into that, I want to share what this extraordinary year has been like.

In my original post - just one month after I started - I was still using the setup as a light, silent laptop replacement. Now I use it in ways and places I never even considered using a MacBook. It’s no exaggeration to say that, this year, the cloud has set me free.


It’s a crisp summer morning. The clear blue sky promises a hot afternoon, but there’s a hint of freshness on the breeze that ruffles my hair. I smile in the dappled shade of the tree and lean back against the rock. My fingers lazily stroke the screen, scrolling through the feature spec, but my mind is a thousand miles away weighing up design decisions for our new product.

I’m constantly surprised how readable the iPad screen is outdoors. Despite being glossy, it’s much better than the anti-glare MacBook Pro screen. It makes working outdoors not just possible, but enjoyable.

A fine mist of tiny droplets falls over the screen and I casually wipe it clean; working next to the fountain is refreshing but I doubt I’d have risked $2000 of hardware here. The iPad’s proven to be a rugged little thing; it doesn’t seem to care if it gets wet and as my files are remote neither do I.

A buzz from my iPhone tells me my build and tests are finished. I pull it out and glance at the push notification, courtesy of Prowl. New test failures. I finish jotting down my thoughts so far into the Pages document I’m using for the spec then swipe back to iSSH. The tethered 3G connection from the iPhone is great, there’s no noticeable lag as I flick to my Vim screen and hit a shortcut that opens and formats the test results.

Hm, the “quick fix” I came up with on the ride here this morning has caused problems elsewhere. I dive into the code for a while, effortlessly navigating around our million line codebase with an instant code search web app I hacked together using python and flask. Happily, iSSH allows me to forward ports through to the iPad, making it easy to securely use HTML for my scripts and helper tools without exposing any HTTP ports to the outside world.

I finish correcting the fix and push it to our main repository. Jenkins will tell me if it passes on the test cluster in due course. I glance at the time and realize I’ve been sitting here for two hours now. I find I can work almost anywhere for two hours at a time, but to stay in one place for longer I need more comfort than a jumper to sit on and a rock for a backrest! Our mid-morning devteam meeting will be starting in fifteen minutes so it’s a good time to go somewhere quieter for the call.

I throw the iPad and keyboard into my rucksack, wade back across the stream and hop onto my bike. A brisk ride later and I’m back on foot, strolling through trees and flowers with my hands-free headset on in a vast landscaped garden that cuts right through Munich. The call is taking some time to set up. As I’m on 3G, I asked the office team to call my mobile number, but one of our developers is at home and doesn’t have a landline in his study, only Skype. An unexpected hiccup - I feel bad; if I were in an office this wouldn’t have happened.

While waiting for him to start the call with SkypeOut, I discover a rose bush and spend a moment enjoying their rich scent. It feels almost like cheating, working in such a beautiful environment, combining meetings with summer walks through the park. And yet, it doesn’t seem to have affected my productivity at all.

I wish the rest of our team could enjoy walking and talking about the last and upcoming week out here surrounded by nature instead of being stuck in a drab air-conditioned box the best part of a thousand miles away.

Then it strikes me: they could. And yet I worked in that office for six years and I never did. There’s a beautiful castle near the office set in vast landscaped gardens; I could have switched to an iPad + server (one of the office servers would have been fine) and spent a day working there, or any of the other beautiful places I loved to be in.

Of course, back then I used a graphical IDE and the iPad didn’t exist, but surely it would have worked with a laptop, too. Well, unless I ran out of power after an hour or two of multi-core compilation. Or it got too sunny. Or it started to rain. And then I’d have had to worry about finding a nice cafe that also has a free space on a large enough table near a power outlet. So many conditions. So much to fear. 

The 10-hour battery life, 3G connection and small form-factor of the iPad + wireless keyboard combination frees me from so much; today I can work wherever I can sit.

A chime from my phone snaps me out my reverie and we begin the call. The wind has picked up, so I mute myself when not speaking and cup the microphone in my hand when I am. It’s not perfect, but the fresh air filling my lungs and the rustle of the leaves in the trees overhead are worth it.

After so much nature I feel like sitting down with a proper seat and table when the call is done, so I cycle down through the park until I reach the Chinese Tower - a beer garden with fantastic benches in the shade. The perfect place to finish my morning’s coding.

At 1pm I leave the park and cut into the city, spending 45 minutes chatting with Matthias from MunichBeta over lunch in a tiny little Chinese place he knows, followed by a visit to Munich’s best ice cream parlour across the street. Matthias lives much as I do, flitting around the city trying out new places to work with his trusty Macbook Air.

This afternoon I’m giving a webinar to prospective customers; I need a quiet conference room and a rock solid internet connection for Skype. My location of choice: Combinat56.

I find a change of scene and some exercise are perfect to ward off any post-lunchtime lethargy and when I arrive at the stylish Combinat56 offices fifteen minutes later I’m feeling refreshed and ready to go.

Running the webinar from the iPad works just fine - Cisco have an iOS app and in any case we’re only sharing slides and dialling in. Usually I just email people the deck and call them on the phone, though; it’s simpler for all concerned.

After the call I catch up with my friends from the Combinat over a cup of tea in the lounge area. You can’t explore on your own all the time; working from home and in cafes is isolating after a while and I enjoy the community here as much as the decor.

While I work, afternoon becomes evening and I sign off from our team chat servers. On a whim, I kick off another round of data analysis on the million song dataset for a side project, then close the iPad and drop it into my bag. Knowing the Linode will keep working away at it, uninterrupted, while I cycle home and enjoy dinner with my wife and children gives me a warm, fuzzy feeling - my tireless robotic assistant, working away out of sight.

Trouble in Paradise

And yet not everything is perfect. I love having my data on a remote server and I’m deeply happy with my indefatigable Linode. Surprisingly, the weak link in all of this has become the iPad. And not just because of what it by design can’t do, but because the internet is moving towards richer and richer web apps and Mobile Safari still feels like a toy browser. 

Using Google Docs has been a pain all year and it isn’t getting any better. 

Having used the fascinating LightTable on the desktop, writing Clojure without it feels like wading through treacle, but Mobile Safari doesn’t have what it takes. I’ve even looked at hacking the engine out of the source and integrating it with the console, for goodness’ sake.

Luckily, splitting my devices between the server and a client means that I am free to change devices whenever I wish. I can work just as well on the twisted remains of my Macbook, a 3-screen office desktop or anything else with a browser, a terminal and a VNC app. My switching costs approach zero. And switch I will.

You see, a surprising alternative has appeared: the Microsoft Surface.

Surface + Linode 512

For a while now I’ve been telling people that Microsoft will become the new cool, the inventive underdog, and I still believe that. Windows 8 may be a huge gamble for Microsoft, but Windows 8 RT is a clear win for me.

Microsoft understands the keyboard. I can start, switch and control apps without leaving the keyboard. The device even comes with one.

Sometimes it’s really nice to have two windows open, especially when using video output to a larger monitor. I think the Windows 8 side-dock idea will suit me very well.

Love or hate Internet Explorer 10, I have every expectation that we’ll see the real rendering engine on Windows RT. I don’t care what they call it, if I can run LightTable and Google Docs without gouging my eyes out, I’ll be happy.

Last but not least, the Metro vibe feels fresh and new and I’m intrigued by Microsoft’s choice to make Javascript + HTML5 a first-class way to develop for the system. I’m already looking forward to hacking my own tiles together to smooth my workflow and simplify my day.

Perhaps in a year’s time I’ll be switching to another client. It doesn’t matter, and that’s the beauty of this setup - its flexibility. For me, though, this coming year will be the year of the Surface.

The Experiment is Over

Last year I started this as an experiment, but it stopped being that a long time ago. Today the entire city is my office - its parks, its countryside, its cafes and its workspaces. I have worked on river islands, half-way up trees and on exclusive rooftop terraces.

My side projects are simpler, more flexible and more exciting now my development machine is connected to the internet by a big fat pipe, 24/7. Want to run statistics over 5 years of Gmail messages? Just do it! No need to worry about how long it takes or whether someone will reset the wireless router part way through. It’s simple. It’s pure.

Feeling free to set off into the unknown, assured that I will find a place I can work, to explore and enjoy my surroundings… no, this isn’t an experiment any more.

It’s liberating, rejuvenating. It’s a way of life.

I will not give it up.

Postscript: Intrigued and looking for more? Tempted to try it out one afternoon? I’ve decided to start a mailing list - I plan to write one or two emails a month about my experiences; the highs and lows of this desk-free lifestyle. I’ll also share more practical tips such as setting up a VPN or keeping in touch with the team. You can join the list here.

Breaking news: Today Allinea ordered me a high-end laptop with a massive SSD for doing CUDA development on. Stay tuned to see how the iPad+Linode combination stacks up against top-of-the-range hardware!

Today the visionary Light Table IDE from Chris Granger reached its $200k funding goal. We’re all excited about using it as an IDE, yet there are three things about the project that excite me even more! Light Table can:

1. Give New Programmers a Reason to Learn Clojure

Chris’ first concept video was met with some criticism that its values are ill-suited to real programming - say, Java enterprise development. These arguments are valid and yet irrelevant. Not being able to visually test and debug APIs that require complex, opaque class types or database access are not reasons Light Table will fail, they’re reasons some new programmers will learn Javascript and Clojure instead of Java and C#.

Having instant feedback and a great visual playground is so much more compelling than figuring out Ant build dependencies. Not all programmers need to pass ObjectMethodFactoryFactory classes around on day one (or at all, but that’s another topic).

2. Encourage Better Coding Practices

Just because writing to a database isn’t code we can happily let the IDE run and re-run for us with different inputs while we work on it doesn’t mean we won’t use an IDE like that. It means we’ll separate code with side effects from algorithms and business logic. Wait, isn’t that what TDD was about?

Having certain great features not available to functions with side effects is a wonderful motivation for writing as much of the code in a functional style as possible and clearly separating code that’s difficult to test from the rest. Who doesn’t want that?

3. Mark a Coming of Age for Functional Languages

I can’t name a tool I use every day that was written with a functional language; the only complex one that springs to mind is Emacs - not exactly an inspirational poster-child. Popular wisdom is that functional programming is great for some niche areas or as parts of a solution but that you can’t create serious, useful software with it.

Light Table is written in Clojure, a Lisp dialect, and it will prove this wisdom wrong, wrong, wrong. Or possibly right! But I doubt that - it works just fine for Emacs, after all. No, the perceptions are wrong and it’s time they were challenged again.

Our world could do with more functional programming. I work in supercomputing and it’s clear that massive parallelism is coming everywhere - and soon. Code with explicit side effects and carefully-managed mutable state can be parallelized so much more easily that it’s not even funny.

I believe we’re entering an age in which functional languages will really blossom - and Light Table marks the start of this exciting new period! Good luck, Chris!

Update: Light Table reached its funding goal, and Chris promises to release the source as soon as is practical (see the comments) - everybody wins!

I love the principles behind Light Table, Chris Granger’s excellent IDE concept based on Bret Victor’s incredible vision. It’s up for funding on Kickstarter now, but it might not make it’s $200k goal:

Honestly? I’ll be happy if it doesn’t get funded.

So is it Light Table you don’t like, or is it personal?

Are you kidding? I love Light Table and I think the world will be a better place for it. I don’t know Chris, but I’ve read some of his code and have a deep respect for him and his work.

Why don’t you want it to get funded, then?

I’ll be happy if Light Table isn’t funded because Kickstarter feels like the wrong thing to do. Chris writes:

I haven’t thought about everything - and I won’t be able to… Light Table is meant to be a platform and it will be open source for that very reason… This is the power of a community full of builders… In the end, the only way to move us forward is for the community to rally behind something.

— Chris Granger, On Concepts and Realities

I couldn’t have said it better myself. To me, this makes the choice for how to proceed with the project crystal clear, but Chris seems to have chosen poorly. There are two options:

You can also get rich playing poker


As the concept video reached 80k hits and the blog post 200k, I’d have been thinking:

Wow, two hundred thousand hits, that’s like - if every one of those bought a licence for $50 then I’d have TEN MILLION DOLLARS and a dream job and would never have any worries ever again! Hooray!

— What I’d have thought, if I were in Chris Granger’s position

There’s nothing intrinsically wrong with this. Clearly there’s a need for a product like this, and if a (finished) version of Light Table for Python existed today I’d drop $100 on it without even blinking. Perhaps even $500 for C++. And I’d be happy to do so.

Chris has chosen this route - trying to build a profitable business - and to his credit he’s also chosen to open source the core of Light Table, albeit in May 2013 after its release. And you still have to pay for a license.

Which brings us to his mistake.

Start a fire

Option 2: Change the World

Instead of putting up a kickstarter page, Chris could have put his source on GitHub, written about the principles he hopes the IDE will enshrine and started accepting patches. It would have instantly become incredibly active; he’d have his work cut out reviewing and accepting pull requests, discussing UI design and principles and making his vision real.

Perhaps it would have been valuable enough to the community for him to go full-time, in which case he could have opened a kickstarter to fund himself as the leader of the open source project. Set a low goal (one month’s worth) and then see what the community wants to pay him to do. There are other options too, such as looking for corporate sponsors, or just doing what he can in his spare time and delegating the rest.

The end result is less certainty for Chris, but much more certainty for everybody else.

What’s Wrong with Option 1

Compare these two projects, for a moment:

  1. Nothing happens for a year while two to three people work together. People get to play with beta release and give suggestions, but every single line of code comes from that handful of minds. Most of the world has to wait for 12 months and then they get to buy a Clojure IDE. Yay.
  2. A veritable explosion of activity as everybody forks the project, tries to get it running for their own favourite language, adds the features they need to make it useful in their day-to-day life. From day one, whoever you are you can find a fork adding support for your language, the better of which are progressively pulled back into Chris’ main repository. Conservatively, hundreds of developers working together. What will this look like after 12 months? Will there be Python support? Ruby support? Java? Integration with other tools? Vim bindings? Emacs? Themes? Yes, yes, yes, and more.

Clearly option #2 is going to be better, both immediately and after 12 months. Trying to develop a community-style project like this with a small team and closed source is a ridiculous mismatch.

Living Costs Money

There are ways for Chris to get paid to work full-time on option #2, but they ask for a lot of trust - a leap of faith that a community will develop that’s willing to support him. It means taking the chance that the project won’t need him and that it will get along just fine without his full-time attention.

What about later on? Chris and team might get rich selling option #1 licenses for years to come; how can they do that if everything is free and open source with an active, vibrant community?

Here’s the choice reframed: either your time on the project is valuable enough for other people to give you money to do it, or it isn’t. You can open source the project and find out directly, or you can attempt to distort your value by creating a false scarcity, only licensing it to people who pay you.

I don’t know if I could make this leap of faith. It’s still the right thing to do.

Your Choice

If you love something, let it go

Do you really want to extract money from people to work on something the world doesn’t need you to? If not, what are you afraid of?

If you believe the world needs your ideas, if you want to spend some of the few years given to you bringing them to life, then do whatever you can to share them and make these concepts real. In this case, it’s clearly to open source Light Table and trust that the future will bring good things.

Don’t be afraid.