Yield Thought

Setting up an iPad+Linode

Fri, 11 Nov 2011 07:34:06 -0500

Last week I promised to share all the details about my cloud development setup - this is how I did it, in ten ~~soul-crushing~~ liberating steps:

Step 1: Get a Linode

I chose a Linode 512, which has been perfect for my needs. You get all phenomenal CPU power and an itty-bitty RAM. Surprisingly, this is enough as we’ll be doing most of our work on the command-line; you can always upgrade it later.

I also turned on the $5 a month backups. Zero-effort data safety for less than the price of a hot chocolate? Yes, please!

Step 2: Configure your Linode

Linode’s interfaces walks you through adding your new node. Pick a region close to yourself - you want to minimize the roundtrip time to the server. I spend most of my time in Munich, so I have mine in London and get a 30-40ms ping, which is great.

I also picked one of their default 32-bit distributions - if you’re not sure or don’t care then just choose the latest Ubuntu and you won’t go far wrong.

After a few minutes the machine will be installed and booted, ready for use! Time to log in and set things up!

Step 3: Build Vim

I found Ubuntu’s default vim didn’t have everything I wanted, but don’t worry! It’s easy to download and build it yourself:

install-vim

Step 4: Install lots of juicy plugins

I honestly can’t remember which plugins I installed and which ones I actually use. My currently-installed list is:

acp.vim
asynccommand.vim
a.vim
cctree.vim
clang_complete.vim
color_sample_pack.vim
command-t.vim
compview.vim
cscope_maps.vim
fuf.vim
l9.vim
NERD_tree.vim
scratch.vim
searchfold.vim
taglist.vim
tasklist.vim

You probably don’t want all of those; I wouldn’t be without command-t though. You can get them all from vim.org/scripts - you might want to install pathogen and get them that way; I just copied the existing .vim folder from my MacVim installation, hence all the cruft.

Note that command-t requires you to build a stub - follow the install instructions in command-t.vim and it just works.

To use the clang_complete plugin, you’ll need clang: “apt-get install clang” should do the trick. I had to play around with it a bit to get it working on my project, which included adding all the -I and -D command-line options to a .clang-complete file in the project directory.

Step 5: Create your .vimrc

Here’s my .vimrc - some of the things here relate to e.g. our internal tracker system and won’t be interesting for you, but it should be clear what most of these things do and which keys are bound to them:

.vimrc

Step 6: Configure GNU screen

I don’t recall where I got this from, but it works well for me:

.screenrc

You’ll also want to create a few aliases to connect to different screen sessions painlessly - my .bashrc contains:

.bashrc

You can probably change screen’s escape key in the .screenrc file too; I never bothered to look it up. Here, ‘s’ or ‘t’ will attach me to my work or personal sessions respectively.

Many people have recommended tmux - I haven’t tried it yet; screen hasn’t given me a reason to want anything else.

Step 7: Configure VNC

install-vnc

Feel free to pick a display that makes you happy but make sure it matches the DISPLAY=:5 line in your .bashrc!

Step 8: Get a super-sweet thin client

I chose an iPad2, (the 16Gb Wifi model). I haven’t tried working over (tethered) 3G yet, but ssh from my iPhone works well so perhaps that’s also an option if you’re often out of wireless contact.

If you already have a tablet that you like, use that instead - all you need is a good SSH client and a good VNC client. On the iPad I use iSSH and Jump for SSH and VNC respectively. I haven’t tried Prompt or Screens, if you already have them then let me know how well they work out.

Whatever you do, get an external keyboard. The Apple bluetooth one is amazing, I can thoroughly recommend it.

You’ll also want an adjustable stand / case for your tablet. Amazon offers plenty to choose from.

Step 9: Configure your thin client

The hardest part of setting up a SSH client on a tablet is getting your private SSH key on there without entrusting it to a third party. I split mine across multiple services and removed it after recombining it; maybe you don’t really care.

If you didn’t add the export TERM line into your .bashrc then remember to tell your SSH client to report its TERM type as “xterm-256color” - providing it supports this (iSSH does) then you’ll get to enjoy beautiful soft shades in your remote Vim sessions - joy!

To begin with I used iSSH’s VNC client, which worked perfectly well but was a pain to switch to quickly. Last week I started using Jump instead and love it, it’s a super-smooth experience with lots of cute little touches that make me smile. Swiping to it with the new iOS5 gestures is much easier than changing the view in iSSH and the next update will make it auto-reconnect after the 10 minute background timeout, making the whole experience seamless all the time.

I learned a lesson: take a bit of time and money and try out various different clients. You’ll be spending most of your working day in them; try to smooth out every aspect of the experience.

Step 10: Enjoy life in the cloud

There’ll be many other packages you want to install - Dropbox and Google AppEngine spring to mind - both can run in console-only modes.

If you’ve followed this far then you should have a fully-connected remote server and tablet client - welcome to the future!

What auto-plugin do you use for vim? Something that can read referred libraries and not just Language standard libraries (boost et al.) perhaps?

Thu, 03 Nov 2011 04:12:01 -0400

I’ve got omnicomplete for python, javascript, c++ etc. installed. It’s ok, but I plan to switch to using clang_complete as soon as I get around to it - you can’t beat the output from an actual compiler!

I swapped my MacBook for an iPad+Linode

Wed, 02 Nov 2011 09:11:00 -0400

On September 19th, I said goodbye to my trusty MacBook Pro and started developing exclusively on an iPad + Linode 512. This is the surprising story of a month spent working in the cloud.

It all started when I bought my first MacBook a couple of years ago. Frustrated by the inconsistent usage of ctrl/alt/option/arrow keys to jump words and screens and lines, I searched for a new IDE. Instead, I found Vim and fell in love. This isn’t another gushing post about Vim-oh-how-I-love-you-my-sweet-darling, but it’s important to the story - as we’ll see in a moment.

Although I like to use Python and GAE for my own projects, at work we write heavyweight C++/Qt code that runs on clusters such as the 200,000 processor Jaguar machine, so most of my time is spent in Linux and a lot of it on remote systems. Typically I’d develop in MacVim locally and run my code in VMWare Fusion or remotely.

One fateful day, VMWare and OS/X conspired to trash my shared filesystem, losing several days of uncommitted code in the process. I was angry.

While dd was recovering as much as it could, I started toying with the idea of giving up on local filesystems altogether. To my surprise, it seemed possible - even plausible.

I just had to try.

The Setup

It turns out you need a little more than just an iPad and a dream, but not too much more:

Total cost: around $800 + $20 per month

I Wandered Lonely as a Cloud

I typically start my day by catching up on the bug tracker chatter, mercurial diffs and other emails with the iPad in one hand while lying on the Combinat56 sofa.

I actually hate the Mail app for this; the stupid animation when archiving posts adds unnecessary delay and the archive button is uncomfortably placed at the top of the screen. More recently I’ve been scanning mails over IMAP with a python script instead.

Next, I lazily swipe to Safari and review my tickets for the day in our web-based bug tracker then return to the keyboard and fire off a couple of emails before settling back into coding - the new four-finger swipe gestures in iOS5 have really improved my life.

But we were talking about coding, which brings us back to Vim.

Vim: My Home from Home

Perhaps the only reason this transition has been so smooth was because my favourite editor / IDE looks and feels almost exactly the same running on an iSSH console as it did running locally on my Macbook. iSSH supports xterm-256color, which means you can still have pleasant colour schemes despite working in a terminal. All my plugins are there, my code-completion, quick navigation and so on.

In short, it’s a seamless transition from my MacVim envionment. If I were developing OS/X apps with Xcode, or used Eclipse or Visual Studio regularly this change would probably have killed me.

As it happens, working in the terminal on a remote Linode is even better than working locally, thanks to the magic of GNU screen.

GNU Screen is Magic

GNU screen is like a window manager for your terminal sessions, giving you multiple tabs, searchable history, activity/idle notifications and - best of all - persistence.

So I fire up iSSH, tap on my Linode connection and reconnect to the already running screen session. All my terminal tabs are exactly where I left them. Other SSH tunnels are still set up. My cursor is still in the same position. The clipboard is as I left it. It’s as if I never left, because for my side projects, I have a different screen session with a different set of tabs and editor instances running. Perfect separation.

It’s hard to overstate how pleasant it is to be able to return to exactly the same session each day; on a MacBook there’d usually be some other distracting programs left that I’d opened in the meantime, and of course any remote connections would have been dropped. At the very least I’d have used MacVim for something else in the evenings. It might be a largely psychological benefit, but it feels as if I can drop back into flow almost as easily as I left it.

The Good, the Bad and VNC

At work we develop a graphical parallel debugger, so I can’t spend all my time in the terminal. For hands-on tests and GUI work I need X. Luckily, iSSH has a workable, if not perfect, solution:

I tap on the VNC connection and up pops the GUI in shocking 1024x768 fidelity and a painful 256-colour palette. I could have 24bit colour, but somehow I prefer the extra speed to beauty. Although it’s a far cry from using a mouse to interact with a traditional GUI program, the on-screen ‘touchpad’ works better than I’d expect. And as it happens, being limited isn’t all that bad:

One good way to evaluate the usability of a program or dialog is to… try to use the mouse with just one finger.
— Joel Spolsky

iSSH’s VNC isn’t quite as bad as pushing the mouse around with one finger, but it does make you consider users with lower screen resolutions, larger font sizes and mouse control that hasn’t been honed by countless years playing Quake and Starcraft. I’d be lying if I said I hadn’t wished for bluetooth mouse support some days.

Maybe It’s a Lifestyle

Today is not one of those days. I unwrap a chocolate croissant, make a fresh cup of tea and settle down to work; a quick hg pull -u && make to start the recompile, then ctrl-x to my editor tab and carry on coding while the rebuild happens. Ctrl-X is my screen’s ‘hotkey’; it defaults to Ctrl-A but on a wireless keyboard that leaves unicode characters in the terminal - I assume this is related to Apple’s support for some common Emacs key bindings in iOS. Strange, but easy enough to work around.

After a few minutes of coding the bar at the bottom of the screen notifies me that my compile has finished; I ctrl-x back, start a sanity test suite running just to be sure nothing horrible was broken overnight then carry on coding again. Compiling on the quad-processor Linode is around twice as fast as inside VMWare on my MacBook was. It’s also completely, blissfully silent. The only sound is the raindrop-like patter of the keyboard as I work. It doesn’t even get warm, a surprisingly refreshing change from the MacBook keyboard!

I swipe to the DuckDuckGo app and look something up in the Qt APIs, then swipe back. It’s becoming second nature, but I still miss a keyboard shortcut for task switching.

After an hour or two of uninterrupted development the UK team wake up and the first instant messages arrive. Thank heaven for the iOS 5 update! I use the imo messenger app, which is fine but of course before the notification center each pushed message would interrupt whatever I was doing with a frustrating popup. Now, the new notifications center behaves much more like a growl notification would; it lets me know and it gets back out of the way again.

I finish the function I was working on then four-finger swipe to the chat app; it’s my boss reminding me about our 11am conference call.

Skype-based conference calls work fine on both the iPad and the iPhone; at least, as well as VOIP calls ever work, i.e. fine after a few false starts while someone reconfigures their linux audio drivers. I appreciate not to having to think about that any more.

During the Skype call my iSSH session timed out in the background; it’s only held for 10 minutes or so. Fortunately a single tap reconnects me, and through the magic of gnu screen I’m back at exactly the place I left off again.

As is always the case, while fixing one bug I encounter another, apparently unrelated one. Time to submit a quick bug report - in the past I took screenshots on the iPad and emailed them to our tracking system, to get around not being able to attach in Safari. Now, I’ve switched to grabbing them and any log files or stack traces with a command-line app on the Linode itself. We use the Best Practical RT tracker (for our sins), which conveniently comes with a perl script to interact with it from the command line. Doing this is actually quicker and easier than uploading via a browser used to be; one command noting the ticket number and the attachments and I’m done.

The Cloud is Always With You

Lunchtime already! I close the iPad and head off with the others. The remote rebuild I left going will still be there; no need to worry about any uploads getting interrupted, it’s all happening in on the Linode, after all.

During lunch I pull out my iPhone and connect to the Linode again; damn, the build failed early on - out of disk space on one of our office machines. The iPhone keyboard is somewhat painful to use, but for rm -rf /tmp/build-2011-* it suffices. I kick off a new build while waiting for dessert to arrive.

Now I’m feeling nicely fed and just a little dozy; to keep awake I move to a standing workstation and set off some longer-running performance tests on a remote cluster. I find I move around between a lot of different working configurations with the iPad; much more frequently than I when using a laptop. I’m not sure why; perhaps it’s something about having a separate keyboard and screen that makes it easier to find a comfortable typing position, or perhaps it’s just the novelty.

The tests will take an hour or so; I close iSSH and review the latest spec / schedule changes my colleagues have made to an upcoming spec, stored in Google Docs.

Oh, Google Docs, why do you hate me so?

In a bitterly ironic twist, the only part of working in the cloud that doesn’t work smoothly is using Google’s cloud-based MS Office replacement. The mobile word processor is a joke, frequently losing the cursor position and sizeable chunks of text. The spreadsheet version is better, but is still a pathetic experience compared to the desktop version. Of course there are no native apps, and the desktop versions of the sites simply report script errors. I’ve even thought of trying to convince my team to switch back to OpenOffice.

Maybe it works better on an Android - no, wait, that’d be anti-competitive behavior and Google would never do something evil, right?

The tests have finally finished and the results are in; I move to the lounge area and connect up to the big HD TV with the VGA cable; something about seeing console-mode VIM in giant HD makes me smile every time.

There are some oddities in the logs; I copy and paste a few snippets to a colleague and discuss them in chat. I’m glad copy and paste work smoothly, although having 3 different sets of buffers (iPad, screen, vim) does lead to confusion from time to time. Idly, I wonder if any of my colleagues have noticed I’ve been using an iPad instead of a laptop for the last month.

6pm rolls around and it’s time to head home for the day. Despite a full day’s intensive use, my iPad’s still showing 15% battery life. I never bring a charger to work and I’ve never needed one. That in itself is a taste of freedom.

At the End of the Day…

Later that evening I pull the iPad out and open up Pages to finish a blog post. The house is quiet; only the distant sounds of the city drift in from outside and mingle with the gentle patter of keypresses on this delightful wireless keyboard.

I started this experiment because I fundamentally believe that most people don’t want to rearrange windows, babysit their own general purpose computers or back up their data. Sooner or later, almost everyone will work like this and I wanted a taste of what that might feel like. I expected to find something that didn’t work, but as the days turned into weeks and the weeks gathered into a month, I found I hadn’t returned to my laptop even once.

I don’t miss the weight. I don’t miss the keyboard getting warm when I’m compiling. I don’t miss its fragility, both physically and virtually. I don’t miss running out of power. To my surprise, I find I am happy. Coding in the cloud isn’t for everybody, but for my workflow it’s a perfect fit and I love it.

After a few minutes the perfect peace is rudely disturbed by the twin jet-engine that is my MacBook fans spinning up to full power on the shelf behind me. I can’t believe I used to put up with this all day and night.

Despite that, I leave the MacBook running - it’s doing the only thing I still need it for on a regular basis.

It’s ripping DVDs.

Discussion links: Hacker News and Reddit

Metagame Productivity Boost: Stats and Charts

Wed, 01 Jun 2011 09:11:00 -0400

Recently I started optimizing my workflow with various scripts and tools. I measured my performance and found it increased a lot.

Some people asked to see stats behind the story so here they are, calculated retrospectively from log files for 5 months before the change and 5 months after it.

4x more productive overnight

I’ve been tracking 3 easy-to-measure metrics:

Bugs fixed
Change sets committed
Number of functions changed

For your viewing pleasure and mine, I’ve tweaked my script to render graphs with the GChartWrapper python module:

I started trying to improve my own performance sometime in mid-January; unfortunately I didn’t keep track of the exact date.

The data are scaled to show bugs fixed per man-month rather than calendar month; from February onward I’ve been in parental leave (Germany’s extremely generous Elternzeit to be precise) and have been working exactly half-time at 18 hours a week.

As a rough measure, let’s compare my performance in Aug-Dec to that in Jan-May, ignoring the fact that I started this scheme mid-way through January and not at the start of the month.

Performance boost: 55.4 / 9.4 = 5.9 times more productive

A changeset can be anything from a bug that was too minor to both putting through the bug-tracking system (such as a message box typo) or a little tweak or extra feature added in passing. Static analysis tends to throw up lots of small fixes that I address with direct commits, for example.

Performance boost: 100.4 / 28.0 = 3.6 times more productive

I started tracking functions changed just out of curiosity; the numbers of tickets fixed and changes committed are open to various levels of manipulation - I thought the total number of functions touched by a change would be an interesting check to make sure I wasn’t just solving all the easiest tickets or committing lots of tiny, pointless changes.

Performance boost: 344.8 / 85.6 = 4.03 times more productive

Hey, is that awesome?

Nothing about the above measurements or calculations is perfect; simply ignoring December artificially depresses the performance boost, as does treating all of January as the post-boost phase. On the other hand it seems incredibly unlikely that working half-time is exactly half as productive as working full-time.

We could argue that the email and communications burden remains the same when working part-time, so we’d expect less the time to be available for productive efforts. Or on the other hand that most of a full working day is wasted anyway.

We could argue a lot of things, but none of these concerns can account for a 400% increase in productivity.

Not only that, but the number of bugs fixes (5.9 times increase) - is a metric of real value to our company. Generally our software director chooses the set of bugs to be fixed for each release and gives them priorities, which means all those bugs had to be fixed and I wasn’t at liberty to pick all the easy ones.

I do like fixing the easy ones, though.

Whether my measured productivity has doubled or increased ten-fold is really neither here nor there; even just doubling your productivity by any set of measurements over a sustained period of five months is incredible.

I’ve also been a lot happier, which is like winning twice.

Wait! There are 3 reasons not to do this!

“I changed my editor bindings, and now I’m a super-hero coding monster. Also, my breath cures cancer. Hooray for Command-T!”

I’m tired of posts like this too.

I want to make something clear: there are really good reasons that you can’t write a commit script, track your performance and expect it to quadruple. There are good reasons not to even try:

Reason #1: I was intrinsically motivated; that’s why it worked

In psychology and the study of education it’s well-known that intrinsic motivation is everything in learning and self-improvement. I already yearned to improve my own effectiveness, that’s why I started doing this. It fascinates me, it calls to me. That’s why it worked.

If you already long deeply to improve yourself, you will do with or without my advice. If you don’t, then drawing little graphs won’t help either. There’s no reason to believe that blindly going through the same motions I did will produce the same result for anyone else.

Your motivations can and do change over time though and maybe reading this will help to motivate you too - I hope so!

Reason #2: never replace intrinsic with extrinsic motivation

It turns out that offering employees bonuses or performance-related pay can be harmful. In some cases - such as programming - an external motivation, such as a bonus, produces measurably worse performance than an internal motivation, such as the love of coding.

Worse still, such an extrinsic motivation will often replace an intrinsic one, damaging performance and job satisfaction.

If you start measuring your performance, you may end up replacing your intrinsic motivation to do a great job with an extrinsic one to make the numbers go higher.

If you start measuring other people’s performance, you certainly will.

Be very, very careful.

Reason #3: these stats don’t measure programming excellence

So, should I ask for a 400% pay rise now? Am I four times as valuable to the company than I was last year?

No and no. I like to think I’m somewhat more valuable to the company now, but not by a factor of two, let alone four! You see. these stats don’t encompass everything a programmer has to do to produce awesome software; they don’t actually try to - they have another purpose.

It’s like running and football: professional players care about how fast they can run 50m. They even practice sprinting during training. It has some value to them. Yet if you ranked all professional football players by how quickly they can run 50m, you wouldn’t find all the top players at the top.

How would a football team composed entirely of olympic sprinters fare? Exactly. Running fast is one tiny part of playing football. Resolving bugs efficiently is one tiny part of writing excellent software.

Stats such as these are useful for your own training, for measuring and improving one specific aspect of your competence but they are worthless other purposes. Don’t use them for:

Comparing developers to each other
Measuring or motivating a team
Hiring new people

It should be clear that single-figure statistics are useless for these goals, but people love graphs and they love putting numbers on things so I’ll state this as clearly as I can:

It doesn’t matter how many bugs you fix per month if your code sucks and the UI looks like a Fukushima control panel after the explosion.

The bottom line is that you can use these stats to measure your own performance increases, but once your goal becomes to increase the stats, you’ve already lost.

Good luck!

Postscript: If you’re interested in seeing my sorry collection of hacked-together python and bash scripts, leave a comment to that effect and I’ll put together a “Metagame: Nasty Hack-Filled Scripts” post for you with all the sources. I’m happy to talk about this on twitter, too.

Edit: Here’s a link to the HN discussion and the Reddit thread for this post.

Work Is Fascinating: The Metagame

Thu, 26 May 2011 06:26:00 -0400

I’ve been bored at work for many reasons at many different times, but three things stand out as real killers:

working on the same project with the same people[1] for years and years,
using the same old languages and tools (statically-typed: yuk!) instead of the new hotness,
being forced to work on maintenance instead of new features, or on small parts of an existing product instead of creating something new.

These are the symptoms of a problem, not the cause, and I think most jobs will have elements of them. But surprisingly it turns out that - for programmers at least - boredom is a choice. Recently, I chose not to be bored. I chose to think one abstraction level higher. I chose to play the metagame.

Me: a case study

I started thinking about programmer performance a while ago. Everybody will tell you that you can’t measure programmer productivity, but this is at best a half-truth. We can, and we should. Perhaps what we shouldn’t do is use those measurements to compare programmers to each other, but we can definitely measure ourselves.

When I started trying to measure my own productivity, it g

ot me thinking a lot more about:

my workflow - I check email, pick a tracker ticket and work on it.
the most repetitive parts of my work - why does it take 10 clicks and 3 windows to commit a change?
the core of my job - I’m paid to solve non-trivial problems, not to sort email and adjust ticket priorities.

It really doesn’t matter what anyone’s day job is; whether you’re working on sporting result clips for Google mobile or maintaining a 20-year old internal accountancy package the metagame is the same. The product may be pointless but the metagame matters to us all. It’s about:

surrounding ourselves with tools and systems that make us more than human, a cyborg-like super-programmer - who doesn’t want that?
automating away the boring parts of work, so that we’re left with the interesting, non-trivial decisions and problems.

When I started I was working on bugfixing before a major release, which is very amenable to this sort of thing because there are lots of small, well-defined packets of work. We were about 9 weeks from release and my typical day looked like this:

Check and respond to my emails
Check the test machines and fix any system or build failures
Take a high-priority bug from our tracking system
Try to reproduce it locally
Track down the cause and write a fix
Commit the fix, update the ChangeLog, resolve the ticket
Repeat from step 3

It was quite easy to measure my own performance in terms of the number of bugs resolved per week. Sure, it’s not a perfect measurement - I could also have factored in the priority of the bugs and so on, but getting an accurate figure wasn’t nearly as important as getting some measure, however rough.

With the goal of increasing my rate of bug fixes motivating me, I started laying into my workflow and adding automation wherever I found an opportunity to speed things up:

1. Don’t waste clicks navigating email

Email checking was clearly not related to my core metric (bugs fixed) but I couldn’t just ignore emails, however much I wanted like to. Instead, I added a bunch of extra filters to brutally cull all the emails I didn’t really need to read, then found out how to get keyboard shortcuts working in Gmail (hint: add &kbd=1 to the URL). Now I was whipping through the emails by pressing [ to read-and-archive for 95% of them, occasionally opening extra tabs to remind me of various actions I needed to take later in the day.

2. Don’t type ChangeLog entries by hand

Next up was the ChangeLog. I don’t know if you keep a ChangeLog at work, but we have done for a while. The idea is that with each commit you log the broad reason for the change, then write a one-liner for each function you’ve changed, along with the filename, class and method. Writing these by hand was incredibly tedious. Rather than dump it altogether (for it had its uses) I invested a few hours in writing a 147-line python script that generated the entry stubs automatically with inline diffs for each function changed. I added a handful of vim bindings to jump between the ChangeLog entries and replace each diff with a one-line description of it’s purpose. Suddenly, writing ChangeLogs was ten times faster even somewhat fun.

3. Yes, I said Vim

It’s no coincidence that I started using MacVim shortly before my metagame trip. Vim is awesome for exactly one reason:

A vim script is exactly the same as the keypresses you use when editing

Writing a script in vim is a simply matter of typing the same characters you normally would to achieve that effect (which is why using hjkl for movement isn’t as stupid as you thought it was). I cannot imagine a lower barrier to customizing your editor. The end result is that vim makes it trivial to optimize all sorts of little common operations; it’s the first editor whose macro functionality I’ve used on a daily basis. It’ll change your life. Learn it. Love it. It’s worth it.

4. Make reproducing bugs trivial

My previous attempt to make my job more interesting had been to volunteer to rewrite our clunky, frustrating GUI-click based test system with a new, streamlined one. Since we use Qt, I added a QScriptEngine which essentially lets you write a small C++ API for your application and then call that from javascript files.

This was a lot faster and less error-prone than manually baby-sitting fragile GUI-based testing as we’d done before, but the real benefit was that I had complete control over the system. Every night it ran tests across dozens of platforms and reported any new failures directly to our tracking system. I started adding extra code to our test-runner (a 1,500 line python script) to make reproducing and fixing bugs stupidly easy. Each test report now includes:

the stack trace and variables for any segfaults
the core files for any segfaults in external programs during the run
all log files, stdout and stderr
screenshots of the GUI should a test timeout
a one-line, copy-and-paste command to re-run the test

Adding this one piece at a time wasn’t a lot of work, but it has saved us hours and hours and hours.

5. Add ssh and bash shortcuts

I spend a lot of time on our test machines, so I added hostname auto-complete to my bash shell, added single-line aliases to change to the test directory, check out an overnight build, view the build and test logs, change to the test user and so on. I learned to love .ssh/config, and so should you.

Saving a few keypresses doesn’t sound like a big deal, but it makes me happy every day when I type:

cs
smoketest -kri 14956

to switch to the latest test directory, update its license file and re-run the bug reported in ticket 14956. I could write an essay just on this:

every single element of cognitive burden removed makes it easier to stay focused on the task in hand

Not having to remember a hostname, or a compiler directory, or keep track of which system I’m on, streamlining all that away makes it trivial to keep thinking about the problem and not the distractions.

6. You can’t have too much Vim

I ended up adding lots of vim shortcuts specific to our systems. It’ll look up and fill in the title of any bug ticket for me. It’ll commit my current changes, automatically taking the ChangeLog entry I just wrote as the commit message and optionally resolve the bug ticket associated with that change using the same message. In one command.

Typing that one feels really good.

Bundling up all the points at which I’d otherwise have to switch to a browser, wait for a page to load, this was accidental but brilliant. Waiting is the death of focus. By wrapping it all together into one fire-and-forget command, I can just say the word and then start choosing the next bug to fix, keeping my momentum and focus intact.

I’m not done

Every week I find new places to tweak my setup or semi-automate more tasks. I’ve started producing beautifully-formatted HTML change diffs for code review on my iPhone, while reclining in one of Combinat56’s Sumo bags with a cup of tea. I’ve added extra notifications when a build fails or a test machine runs out of disk space.

I will never be done, because I can always keep moving one abstraction higher until I’m purely working on the most interesting problem of all: how much of a programmer’s job can be automated?

A nice side-benefit is that much of the code I write to automate things is not production code; I can write it however I like, using whichever languages and tools take my fancy. Yet it all goes towards making me - one day - into an extraordinary programmer.

Playing the metagame: advice

After working like this for six months or so, the following pieces of advice seem like the most important to remember:

Don’t be afraid. Spend time improving your efficiency instead of hacking away with a blunt axe. There can be a lot of pressure “just to finish this first”. Resist it. Take just one hour and use it to write a script to help you in some simple way, or improve your bash aliases, or your ssh config, or your mail filters. Once you’ve seen how much impact it makes, you’ll feel a lot better about doing it again.
Semi-automation is better than full-automation. It’s easy to get carried away and try to write your own email client, or use bayesian filtering to answer all your emails. Don’t bother. Write the simplest, smallest thing you can to speed you up and then get back to work. If it helps, you’ll find yourself building on it naturally over time. It’s better to have one-key read-and-archive than a custom email client for your iPhone.
Keep scripts specific. Don’t try to over-generalize them and solve everybody’s problem; that’s not going to be practical on the side. Make your scripts specific to you and your workflow. If they’re great, you can extend them to the team. If they love it, you can extend it to the world and put it on GitHub. But do it in that order.
Use vim. It feels weird at first. It takes a couple of days to get used to. Set it up with nice plugins and themes right away; after a couple of weeks you’ll never look back.
Optimize the things you hate, not the things you love. The goal is to build up scripts, commands and systems that feed you a flow of meaningful decisions to make and interesting problems to solve, while automating away the dross. Resist the temptation to start by optimizing programming or code-writing; attack all the distractions first.
Never let yourself wait for something. If you catch yourself typing a command then waiting for it to finish before you can type the next one, then that’s an ideal place for a script. Waiting will kill you; make the computer wait for you. For some tasks, like compiling, write scripts that’ll automatically launch the program when compiling finishes, perhaps even re-running the test case if possible. Make your computer notify you when it’s ready, audibly or by throwing something up on the screen. In the meantime, you can be reviewing code or looking for a second bug to start work on, or taking a walk with a cup of tea. Anything but HN and Reddit!
Write in all the fun languages you can’t use at work. I can’t recommend python enough for this sort of thing, as it’s huge supply of libraries combined with pleasant syntax make life a breeze. I guess Perl would be great too. But get used calling external programs and input / output streams and you can write in Node.js or whatever interests you most this week.
Measure your performance. Write a simple script to measure the number of commits you’ve made. Or the number of lines you’ve changed. Or the number of bugs you’ve fixed. Or the number of customers you’ve helped. Just pick something that’s trivial to measure even if it’s far from perfect, and start. Add more things whenever you’re afraid you’re over-optimizing or gaming the measurements you’ve got so far. Seeing your own performance climb gives a real sense of satisfaction that might be otherwise missing in your daily work. Even if you don’t enjoy or value the internal actuarial calculator you’re writing, you should care deeply about making yourself a better programmer - this will stay with you your entire career.

That’s it - print this list out, stick it next to your monitor then close the browser and start playing the metagame! You weren’t going to do anything more productive in the next hour anyway, right? ;-)

Update: just in case you’re not heading off to work, the HN thread is here and the Reddit thread is here.

[1] Disclaimer: I work with absolutely first rate programmers, people I learn from every day and yet there’s still an attraction to meeting new people too; I shudder to think what it must be like to work with dull people for years and years…

You're On The News

Tue, 11 Jan 2011 10:31:00 -0500

While visiting BuddyCloud Towers over the Christmas break I finally sat down and started something I’ve wanted for a while: up-to-the-minute notifications when one of my posts is submitted to Hacker News:

Timing is everything: the HN newest page is a cruel and capricious master; there are lots of posts being submitted and not a lot to distinguish between them. If you take the time to respond to comments early and add your vote to that of the original submitter, it gives your post a much better chance to be read.

The window before dropping off the all-important new submissions page? 30-60 minutes, give or take. Google Alerts might as well be sent by snail mail.

Other times, an old post is resubmitted (as RiderOfGiraffes was recently kind enough to do for one of mine). When that happened, I wanted to reopen a closed poll, update some old links and join in the discussion. Luckily Mat emailed me when it hit the front page but I could just have easily missed it altogether.

So I hacked together http://youreonthenews.appspot.com/ to solve the problem. It’s a simple two-page app on a GAE stack that does just what it says - emails you when a post from your domain is submitted to Hacker News.

Give it a go and let me know what you think, what you’d like to see and what doesn’t work!

Endlich auf Deutsch: Find A New Game

Tue, 04 Jan 2011 10:49:00 -0500

I’ve made a first attempt at internationalizing http://www.findanewgame.com/ - if any of you are based in Germany or speak the language, I’d appreciate your feedback!

Ich habe (endlich) http://www.findanewgame.com/ auf Deutsch umgeschrieben - falls ihr Zeit und Lust habt, würde ich mich über Kritik und Anregungen sehr freuen!

One of the many reasons i18n is harder than you thought…

Mon, 03 Jan 2011 03:46:18 -0500

How it should be

Gaah!

One of the many reasons i18n is harder than you thought…

$37.91 from frontpaging HN and Reddit - what now?

Thu, 04 Nov 2010 03:53:00 -0400

A fortnight ago I wrote 10 Embarrassing Flaws That Made My ‘Weekend’ App Possible, which was about my two-week project to recommend new board games. As it turns out, that post was picked up by Reddit and HN a couple of days later in a big way, bringing in over 50,000 hits in a matter of hours. This is what that looks like:

At peak around 4,000 hits per hour were coming in, or just over 1 per second. The effect was pretty noticeable on twitter and in my inbox, too:

About an hour into this it suddenly struck me that this would be a very good time to add the Amazon affiliate links to findanewgame that I’d been meaning to. I was in a rush, so I just added a link to the search page, something like this. I wanted to link directly to the products but hadn’t got around to implementing the ItemSearch API yet. I didn’t have time! The site was getting traffic:

Maybe 12% of the blog readers checked out the findanewgame site and some of them bought games! It was working - I’d helped people find cool new board games! By the way, if you’re one of them then I hope you’re having fun with it - I’d love to hear how it’s going.

Fourteen Days Later

Traffic from HN and Reddit doesn’t really stick around - life returned to normal pretty quickly:

The findanewgame site itself was actually submitted to reddit on its own a few days ago (which I completely missed); here’s what its traffic looks like now:

Strangely, most of the traffic to the game site is ‘direct’, which can mean almost anything in analytics. Privacy-aware browsers, links sent thorough offline email programs, who knows? Not Google, that’s for sure:

So 100 Facebook likes drives less than 100 visits? Wow. Is that normal? I don’t know. Maybe I should care, but there’s something more pressing on my mind at the moment: I broke something.

I Should Have Used A/B Testing

You see, the Saturday after the post went big I made some more time to work on the site. Top of my list was sorting out the Amazon links, which I did. Now most games link directly to the product page - saving my visitors an extra click. Great! Great? Well, here are the Amazon stats. Can you tell when I made the change?

Yeah. That’s a lot of 0.00%, huh? Actually I deployed the change on the 23rd, which enjoyed a bumper 4,48% conversion rate. I thought this was fantastic, until the next day. And the next. And every day since.

What happened? I’ve no idea. People are still coming through to Amazon, so the links can’t be broken. They seem to be linking to the right products. For some reason, though, nobody’s ordering games now. Even with a 1% conversion rate I’d expect to have seen 2-3 orders since then.

If I’d been A/B testing every deployment, I’d know whether this was something I’d done or not. I actually can’t imagine why every deployment isn’t rolled out via an A/B testing process. Does anyone know of a really slick way to integrate this with the AppEngine workflow?

Choose Your Own Adventure

Where should I go from here? There are several options:

Roll back the Amazon link changes - maybe people will start placing orders again; it’s a quick change to make, but it’ll take a while to see if it has any effect.

Add A/B testing to my workflow - I’d love to push every deployment into an automatic A/B trial and then later decide which versions to keep and which to rework. Do you know of anything like this for AppEngine?

Post recommendation links to forums - I don’t know anything about marketing, but the stickiest traffic I get seems to be from forums. I’ve recently added short urls; I can spend a few hours visiting board game forums, reading the recommendation posts and using findanewgame to offer suggestions.

Make it easy to create and share lists - this has always been a goal, because sharing things we like is fun and because Christmas is coming up. At the moment you build up a list of games you want to play; I could make this easier, then let you email yourself or someone else the list, share it on facebook or whatever. Perhaps this option could also be called ‘market the site socially’.

This adventure is fun to do and even more fun to share, so which two tasks are should I do next? Vote below - I’ll do whatever wins and write about it again in a couple of weeks!

Browser not showing the poll inline? Try http://polldaddy.com/poll/4033559

Update: Hacker News discussion

Users Who Can't Buy, Customers Who Don't Use

Wed, 03 Nov 2010 03:58:00 -0400

Patrick’s recent business of software roundup included a fascinating paragraph about classifying founders (and hence the applicability of their advice). Apparently Jason Cohen partitioned people like this:

Patrick explains it better than I can:

There are two competing motivations for people who start software companies: wanting to maximize one’s financial outcome as quickly as possible (you want to be Rich) and wanting to sculpt the perfect personal niche in which you will be respected and enjoy your day-to-day work (you want to be King).

The Rich vs King concept was first floated by Noam Wasserman in 2005; if I’d read it earlier it’d have saved me reinventing the concept as Emperor Palpatine vs Han Solo, but there you are.

So Am I B2B or B2C?

Everyone likes to talk about being Rich or being King (or being both), but nobody talks much about the difference between B2B and B2C. This is odd, because it’s more important. Patrick simply says:

There are two fairly well-known markets for software: B2B (business to business) and B2C (business to customer)

For a long time I wasn’t really clear about the separation. Some cases are obvious - Foursquare is B2C and Visual Studio Professional is B2B - but how do I evaluate ideas? Is productivity software for dentists B2B or B2C? What about software to assist translators? Or teachers? What about an easy-to-use profiling tool?

This morning it finally became crystal clear: it’s B2B when the user is not the person making the purchasing decision.

This is the only distinction that matters, but it really matters. Your entire sales pitch and cycle look very different if your customer and your user are not the same person. You’ve got to make the users happy enough to push for a purchase, but you’ve got to reassure, hand-hold and sell to the person with the money. You’ve got to know who your users are and who your customers are, because they might not be the same people.

Be sure. Decide. Any confusion about this will kill a product faster than you can say but-I-A/B-tested-everything.

Postscript: here’s a link to the Hacker News discussion

What, more tests are always the best way to improve my product?

Fri, 29 Oct 2010 06:57:00 -0400

Increased code coverage through unit testing can decrease the number of flaws in the final product, but that doesn’t come for free. Development time is spent working on and maintaining those extra tests. Sometimes the product would have been better for the users if that time had been spent on:

Design
Features
Releasing earlier and more often

More time spent writing unit tests does not imply the product will be better for it.

Everything must be balanced against the product. There’s some level of testing that is optimal. After that point, pushing for higher code coverage isn’t ‘professional’, it’s criminal overengineering.

Discussion on Hacker News

Dear TDD Kool-Aid Drinkers

Wed, 27 Oct 2010 08:56:00 -0400

Show me just one shred of empirical evidence that developing with TDD results in better products. Yes, I’m looking at you.

Seriously, though. Is there any evidence? If not, why not?

Update: shreds of evidence

John Goering provided links to some studies - thanks, John! The preferred study seems to be this one by Microsoft. IBM summed it up like this:

There was a meta-study done in 2006 by ITEA that compared the results of several other studies. Table on page 13 sums it up, which I’ll shamelessly reproduce here unless someone asks me to take it down:

Note: I’m not sure what the ‘Other’ column is based on here.

There seems to have been two approaches to the difficult problem of testing development methodologies:

Measure what happens when real teams use it on real projects and draw comparisons with previous, albeit different projects
Set undergraduates programming challenges and measure their performance

Neither approach attempts to address the more difficult, yet important challenge I originally posed: will using TDD result in a better product? Better can be defined as ‘successful in the marketplace’.

At present the conclusion seems to be “insufficient evidence”. I’m encouraged that people are really studying this sort of thing; it’s a shame the results don’t feed through to the development community as a whole more visibly.

Where did you move to Germany from? Was it hard to move?

Wed, 20 Oct 2010 09:39:43 -0400

I came here from the UK in 2005. I didn’t speak any German at the time. It was a lot of fun - I picked up the language quickly and really enjoy the pace and quality of life over here.

Sorting out the tax and so on went on for over a year before my UK employer decided the simplest thing was to set up a German subsidiary and employ me through that instead!

Being an EU citizen anyway makes everything very easy; I have the same rights here as any German citizen, including health and unemployment insurance, pension and so on. Except I don’t get to vote in the national elections, which seems fair enough!

10 Embarrassing Flaws That Made My 'Weekend' App Possible

Mon, 18 Oct 2010 16:49:00 -0400

Every few days on Hacker News there’s a popular “Ask HN: Review my weekend project” post. Without fail the apps look polished and complete and are almost always interesting. I used to feel humbled by the awesomeness of someone who turned out an entire project in a weekend.

I shouldn’t have - and neither should you. 48 hours is an eternity. What could you do with 2 hours an evening, every evening for almost a month? Today I’m writing about what I did with that time, spread over the last few weeks. If you want to see where it’s up to now, take a look:

It all started with a simple idea, which steadily grew into an interesting, usable website despite at least ten horribly embarrassing flaws - or perhaps even because of them.

Day 1 - The Idea

In Germany there’s an amazing board game culture that I’ve taken to since I moved here. One Monday, a couple of weeks ago, I found myself browsing the reviews on the excellent boardgamegeek.com trying to find a new game that’d be fun to play with my friends and family.

They’ve got this huge database of games and ratings people have given over time and I wanted to do some simple data mining on this to suggest games similar to those I enjoy that I might also want to play.

I didn’t have any plans to make a product or website at this stage. I was just curious as to how easy and effective it would be. So began a fortnight of messing around in my favourite language - python - of horrible hacks and embarrassing mistakes, yet at the end of it a fun board game recommendation site was live and people were using it.

Here’s how it happened.

Day 2 - Embarrassing Flaw #1: Parsing HTML with string.split

In a spare hour in the evening I pull up a python terminal and start trying to actually get some of the ratings data. I start by scraping the HTML pages with urllib2 (don’t forget to use a browser’s User-Agent) and parsing them with, well, this is embarrassing. With string.split. In my defence I spent maybe 15 mins playing around with BeautifulSoup, but all I needed is a couple of chars and hey, string.split and a regexp works first time.

I know. I’m a terrible person. I should know and love BeautifulSoup already. But, hey, it turns out I don’t. If I’d played around with every piece of technology I should be using whether I felt like it or not, I’d never have got anywhere. Maybe I’d have ended up writing half of an AbstractHTMLScrapingWrapperFactory and then lost my will to live.

Sometimes YAGNI (you ain’t gonna need it) can be applied to your own professional development in the name of doing something cool.

Day 3 - Embarrassing Flaw #2: Using the python prompt as my IDE

The next day I discover there’s a well-hidden XML API for Board Game Geek; just what I need for grabbing some ratings data. This time I’m not just trying to scrape a handful of game ids from static pages, but to parse large amounts of varying data. This is the point of diminishing returns for string.split - ugly hacks are fine, but like all good drugs you have to know when to stop. I pick up the excellent xml.dom.minidom module and use it to sample a few thousand game ratings to play around with.

Now I’ve got some data, I want to see if there are any correlations in the ratings between games - I expect that someone who rates chess highly probably rates games similar to chess highly. I love python because it has everything built in, including numpy. My entire correlation function was:

def correlation(game1, game2, ratings):
    rs = ratings_for(game1, game2, ratings)
    x = numpy.array([a for a,_ in rs])
    y = numpy.array([b for _,b in rs])
    return numpy.corrcoef(x, y)[0][1]

Actually, that top line was added later. You see, I was coding directly into the REPL, the python prompt. If a bunch of commands yielded something useful, a building block I wanted to use again, I stripped them out of the terminal history and pasted it into my (single) source file for future use.

I guess sometimes YAGNI even applies to your editor. An editor forces you to write functions backwards - name and parameters first, contents later. Writing in the REPL was the right way around - do something to get at interesting data and then codify it into a function. I never had to wonder what a function should be called or which arguments it should take.

You wouldn’t catch me doing this in my day job, though. Code completion, find references, these are things I can’t live without when maintaining a big codebase. But writing a little tool from scratch, that I’m exploring as I go? Absolutely.

I run correlations between all the games in my small sample set and see correlations of around r=0.6 for some games - this isn’t huge, but it’s interesting. There’s only so much you can do with a massive matrix of correlations, though. I realize I need a way to visualize this stuff.

Day 4 - Awesome Decision #1: Use Someone Else’s Code

I begin the evening hunting around for a good graphing library that I can actually get running on my MacBook. I spend at least two hours just doing this, going down dead ends, fighting with MacPorts, debating whether graphviz will be good enough or not and so on, until I finally stumble upon the divine NodeBox.

NodeBox has a dot-like graph layout library, which is pretty close to what I need, while integrating nicely with python. The documentation is good enough to get me going and to produce a graph layout of the 30 most popular games in my sample, with edges between those with reasonable correlations, shorter for higher correlations. You can see the trivial python code used to generate this on the right:

I thought this was pretty great - the games (vaguely) fall into categories that make intuitive sense! I immediately stopped work and showed this to everyone within reach.

Doing this reminded me how much better visualization is than assumption. It’s prettier, too.

Day 5 - Embarrassing Flaw #3: Making a stupid business plan, then ignoring it anyway

While thinking about this on a plane to England, my mind makes the natural leap from my motley collection of python functions to turning down acquisition offers from Amazon and being invited to talk at Davos. While musing on how to get from here to there, I take a liking to a simple little opportunity: write a site that recommends board games to people, then rake in the money through referrals.

Later that night I check out the Amazon referrals scheme and make a quick back-of-the-envelope calculation:

Referral rate for mere mortals: 4%
Average price of a game: $25
Referral income per game: $1
Adwords cost to drive traffic: $0.25
Conversion rate required: 25%

Boom! Shot out of the water. I see no way this could ever become a money maker.

But, I still want to do it. It’d be useful, for me if for nobody else. And hey, I can throw it up on AppEngine for nothing and my friends and I can use it. I gently pack away my dreams of grandeur and decide to make this into something fun.

If I’d spent more time on a business analysis, I’d doubtless have turned up things like alternative traffic aquisition models, viral this and facebook that. If I did it for long enough maybe I could have convinced myself this was a profitable opportunity.

Instead, I accidentally discovered something more important: I wasn’t doing it for the money. If I had been, I’d probably have gone down a different route or given up by now.

Days 6 and 7 - Embarrassing Flaw #4: Being worse than a one-line algorithm

I spend a couple of evenings lounging on a comfortable sofa in front of the fire in cold and rainy England, writing code to calculate polynomial fits between the ratings for pairs of games. I use the this and their correlation to predict a score for one game given a set of ratings for other games. With each function I’m building myself closer to being able to give my favourite games and get a list of recommendations out.

One dark and stormy night, I reach that point. The final goal. Does it work? I test it against another sample from the DB. It predicts 84% of the games correctly! It works! I’ve done it!

Suddenly, doubt strikes. Lightning didn’t, but it would have been more dramatic if it had. I suddenly ask myself: what’s the baseline here? Anxiously, I write a one-liner that just recommends the most popular game all the time. It scores 88%. Oh.

Days 8 and 9 - Awesome Decision #2: Do nothing

I spend a couple of days not working on the problem, but I do think about it every now and again. Despite the setback, the correlations seen in the graph make me certain there’s some merit to this idea. On the flight back home inspiration strikes and I jot down the basics for an alternative algorithm. It seems less scientific, but it might just work.

In hill-climing terms, this was a clever move. I didn’t realize it was a clever move at the time; I just didn’t feel like writing any more code for a while. If I’d put myself under the pressure of creating a sure-fire money maker, I might have tried optimizing the algorithm I had, tweaking this and that, findally climing to the top of a very little hill in the solution space.

So, +1 from me for giving up when the going gets tough (as long as you come back later when the tough’s not looking).

Day 10 - Embarrassing Flaw #5: Brute force is my algorithm of choice

Back at home again, I code up the new algorithm. It’s not very complicated and I can re-use most of the supporting functions built up already. I try it against the old one and against the baseline.

90% accuracy. Better, but percentage correct isn’t very useful at these levels. It’s predicting one game in ten wrong, which seems like too many to base a purchasing decision on to me.

There are a lot of ‘magic’ parameters I can tweak, but where to start? I briefly consider using a GA or NN (both pet interests) to optimize the algorithm, but frankly I don’t want to learn about some guy’s NN framework - I just want to make my algorithm a bit better. So, slightly curious as to the result, I write a few loops to brute force it. What do you know? It completes before the end of the universe. In fact, before the end of the day.

I’d have loved to have had some cool, cutting edge technology like NN or GA optimizing my recommendation algorithm. It’s so much sexier. But, you know, just trying all the combinations actually worked out great. I guess brute force is its own cool.

Now I have 94% accuracy - the failure rate has almost been halved. But how does it stack up if we start removing people’s favourite games? After all, it’s easy to recommend a game everybody loves, but I want to recommend games people don’t already have! I code it up and try it out.

94%, 95%, 92%, 94%… it’s looking good even when it starts recommending oddball games way down the top 200 list, although it does drop off eventually. I try it on myself. It recommends games I already wanted to try out. I try it on a few people I know. It seems reasonable. I’m excited! I want to show it to my friends and family and let them try it, but I can’t because it’s just a python script.

Time to change that.

Day 11 - Embarrassing Flaw #6: My website is a popular free CSS template with some badly-written JS on top

Two things I’ve learned about writing little web sites:

I have no graphic design skills
A site with a cute template is a lot more fun and rewarding to work on than a blank page with input boxes

With those in mind, the first thing I did was hit up the free CSS template sites to find something not too hideous that I could use as a basic template. I knew I wanted a logo, a couple of input boxes and space for a nice list of game boxes. Almost immediately I found one:

I tweaked it a little to suit my needs, yet I was still ashamed, knowing that anyone who’s seen this template before (and it’s one of the more popular ones, apparently) will instantly know that I have no CSS skills and am too lazy to subcontract to a proper designer.

I remind myself that pride is another thing I Ain’t Gonna Need and get on with it.

As an aside, I’d recommend Google AppEngine in a heartbeat to anyone who already knows some python and wants to host a little side project:

Great basic API - you can go from zero to working page serving DB requests in half an hour
It just works - creating a new app is easy, deployment is single-click
Generous free limits - I’ve never run past the quota yet
No performance worries under load - unless you do something stupid to your DB, I guess

Day 12 - Embarrassing Flaw #7: It takes me all evening to get autocomplete working, and it still doesn’t work properly

Now I have a site that looked ‘real’, but I wanted it to feel right. I’ve two text boxes for game entry and I really need auto-complete on the game names. I mix in JQuery and JQuery UI for the autocomplete and instant search - neither of which I know much about. In fact, I’m such a n00b that after spending all evening reading the JQueryUI docs and messing around with select vs close events my autocomplete popup and form event handler still don’t work properly - pressing Enter too soon will result in your game being silently ignored. But, hey, it’s close enough.

In the end, spending the time on the behaviour and the looks were worth it. Behaviour is fundamental - do you choose single items from a list, or auto-complete? Is it easy? Is it fun? These choices are somehow the core of a user experience.

Even if I change the visuals for a proper design someday, the user experience will stay pretty much the same - and I wouldn’t have figured that out without my free CSS template and hard-won-yet-buggy autocomplete implementation.

Day 13 - Embarrassing Flaw #8: My NoDB implementation

The rest of the site was really simple. Too simple, in fact. See, my python script didn’t have a DB, it just used pickle to load the data from a file. So my Google AppEngine project didn’t have a DB either. Take that, NoSQL guys! I’m NoEverything!

This is really shameful. It would be measurably better to use the DB. There’s a delay of several seconds whenever someone goes to the page for the first time since Google decided to kill off my instance - luckily the instances live a long when the site’s under load. It stops me using a larger sample set - even though this would give better results - because doing so forces the process to exceed the soft memory limit, meaning it gets killed at the end of each request.

But, you know, if I’m going to put the DB in, I should get the GAE app to fetch the data too, rather than doing it in batches by hand. And I should get the data in better formats to solve X and Y and Z. And at this point, I could easily spend several evenings making it work right. Call it a week.

Yet even without all that, it’s actually pretty good.

I’ll change it soon, sure, but in that moment of analysis paralysis I decide to leave it as it is and hope that nobody ever finds out.

Day 14 - Awesome Decision #3: Begging for feedback

At this point I have a single web page that:

Invites you to enter one loved and one hated game into auto-completing boxes
Instantly updates to show you a list of recommended titles, along with game box images

It doesn’t do anything else. I have so many big plans - the classic social media buttons, loving / hating multiple games, saving the state in the URL, creating sharable links, showing mini reviews of each game…

I don’t do any of those. Hundreds of blog posts telling me “If you’re not embarrassed by version 1, you didn’t release early enough” have made their mark on my soul. Instead of making it better today, I post this to the Board Game Geek recommendations forum:

Hi all,

I always come to BGG to find a cool new board game to play, 
but I end up spending hours reading reviews in forums and 
trying to guess at what would suit me, my wife and my friends 
best. So when I saw the XML API, I just had to write this:

http://findanewgame.appspot.com/

Tell it which games you love and one you hate, and it shows 
you a selection of games you'll probably love. It certainly 
works for me, and after testing it on a ~2000 user sample of
the BGG database the algorithm proved to be 94% accurate at
predicting which games a user would rate highly, which I
found pretty stunning.

It's just a bit of fun at the moment, but I'd love to hear
how useful it is / isn't and which other features you'd like
to see.

Enjoy!
Mark

Immdiately I begin refreshing the page compulsively, waiting for someone to reply.

They do.

It turns out that getting in touch with a community for my app was easy. This makes me think that if it’s not easy, then maybe you’re developing an app for an imaginary group of people who don’t exist.

Days 14-19 - Embarrassing Flaw #9: Attention addiction

That week the main page was hit 1500 times and the post racked up over 100 replies containing bug reports, suggestions and feature requests. The game was on! Every evening and lunch break I added a little feature here, tweaked the javascript there. The hardest bit was not working on it the rest of the time!

Every piece of feedback, every happy post fed a warm glow of pride and smug self-satisfaction. A little too much, frankly. Still, it made being responsive to people very easy, and that seemed to make people even happier. By the end of the week I completed the last major feature addition people had been asking for. Now the site was - as far as I could tell - pretty useful.

Next time, I won’t worry at all about launching a buggy website. People tell you about bugs and this gives you all the motivation you need to get in there and fix them! And if a site has any value at all they’ll try it a second time, so you can’t actually lose. Unless you’re launching a bank, I guess. Or a nuclear reactor. I promise not to do that.

Days 20-27 - Embarrassing Flaw #10: I burned out

By this point I’d been spending most of my spare time working on or thinking about this little site for the last two weeks, and suddenly I found didn’t want to do that any more.

The forum post slipped off the main page, the stream of visitors slowed to a trickle (mostly through email referrals and direct hits - presumably some word of mouth).

I decided I’d had enough. I took a break. I went for walks and spent time with my friends and family. I met interesting people and did interesting things.

I guess if I’d thought I was in a race to release, if endless riches were waiting just out of my grasp, I’d have forced myself through this phase. Or if I’d been burning VC money, my runway rapidly disappearing, the date of failure already marked on the calendar…

But I wasn’t in either of those situations. I just let it rest, knowing I’d come back when it was time. Still, the thought of just leaving it there without somehow pushing more, doing more, that’s was embarrassing. But necessary.

Here I Am

After around a couple of weeks completely ignoring the site, it’s still getting hits from somewhere - mostly email referrals as far as I can see in the logs:

My goal is to put a basic, yet fully-functional app out there and see at least *one* end-to-end conversion. That plan looks like this:

Add a short paragraph at the top of the page explaining why you would use the site and how to get started - it’s not obvious enough as it is
Show more information about each game on click, including a ‘buy now’ link to amazon and a description / mini review
Add conversion tracking

There’s also a ton of things I really, really should do, but won’t (yet), like:

Add a ‘share this list’ section, offering tweeting a link, sending to your facebook wall or emailing it to a friend
Internationalization - especially the German market (board games are huge in Germany). This needs translations of the game titles and redirection to amazon.de instead of amazon.com. I’m sure you can do this with their API.

Once I’ve got a minimal yet complete site up, I’ll find out whether it’s something worth spending more time on. Either way, it’s been fun! And without really realizing it I too have a done a ‘weekend’ project I could submit to HN, just like all the cool guys do ;-)

Update: this article is front-paging both Hacker News and Proggit, so I’ve added some extra text at the top and linked some (but not all) items to an amazon.com search page. I guess I’ll get back to plan A and proper referral links when the dust settles a bit!

Two Things I Learned On The Way

So far, I’ve learned two things from this experience. The first is that although I created something ~~filled with~~ consisting entirely of embarrassing flaws, without taking those shortcuts and without making those mistakes I would never have come as far as I did.

Just because your code is horrible doesn’t mean it’s too early to show people the end result - they won’t care what your code looks like.

The second is how easy this sort of thing has become. I did this over a few weeks without any special skills and with a full-time job, a social life and a family. Most of the time was just exploring a fun side-project; putting it online was the least time-consuming part.

I Did This; You Should Too

How many half-finished projects litter your hard drive, never to be seen by another living person? What a waste! These days it’s so quick to get a half-finished project online so that other people can tell you whether it’s useful or not that it’s almost a crime not to give them the chance.

Take your next project (or even better: one of your old ones) and just drag it, kicking and screaming if necessary, onto a server. Send me a link when you get something horribly embarrassing live for the first time; we can laugh about it together and maybe - just maybe - it’ll become something amazing.

HNer? Check out the Hacker News discussion of this post. Or do you prefer Reddit? Either way, follow me on twitter and tell me about your next project…

Postscript: You can try the site for yourself here: http://www.findanewgame.com - have fun!

Newspapers: How To Save Your Doomed Businesses

Mon, 11 Oct 2010 06:59:00 -0400

Every day a big fat newspaper lands in my letter box. I don’t even know why; I think a friend had their trial subscription moved to our address while they were on holiday and are having trouble canceling it. Anyway, every day I lift it out and dump it in the lobby for or someone else to read. Most days I don’t even read the 72pt headline.

It’s not that I dislike reading. I love reading.

It’s not that I dislike paper. I’d love to read more on paper instead of a screen.

It’s not even that I dislike their adverts.

It’s this: they’re sending me the wrong words. My interest in the words they print is exactly zero.

There’s at least one man who knows exactly which words I like to read: Marco Arment, the Instapaper founder.

For those who don’t already know, Instapaper works like this: every time I see an article linked on HN, Reddit, or somewhere else (I guess that might happen) and I want to read it when I’ve got time, I hit the Instapaper ‘Read Later’ bookmarklet. The text is then cached on their servers, pushed to all my devices and later on when I’m sitting squashed between giant sweaty footballer and unhealthy cough woman in the underground I can read it on my iPhone.

It’s a wonderful service, and the newspapers should steal the idea immediately. I have never subscribed to a daily paper in my life, but if my daily paper contained the articles I’d bookmarked the previous day, complete with images and nicely fomatted, well, that would make my life more awesome. I’d pay for it at once.

They can continue to mix in their top journalism stories (even if I ignore them, the paper costs them essentially nothing) but now they can replace all the filler that comes after page 2 with my personalized content.

They can get rid of the tiny font and huge form factor and make reading a pleasurable experience - they no longer need to pack as many articles into one paper as possible in the hope of providing me with one that isn’t utterly worthless; now, every article is a hit.

They can use my history of bookmarked articles (and everybody else’s) to trivially predict recent articles I’d love to read but haven’t discovered yet. They could give me a browser plugin to make sure they exclude pages I’ve already visited and either read or decided not to bookmark for a totally painless service.

They should do all this, because:

HN Monthly’s popularity suggests even technophiles still pay for printouts
Instapaper’s popularity shows people love to read articles offline and not while they’re busy trying to get real work done
They already have awesome global print and distribution networks, advertising contracts, market penetration, paying subscribers and brands
Their brand strength will make it much easier to push standardized self-register revenue-sharing deals to the blogs who’ll be providing this content. Bloggers crave attention; who wouldn’t register their content with a prestigious newspaper in exchange for a share of advertising revenues and nice stats about where their blog is being read? Don’t forget the chance to appear on their top blogs lists for even more exposure!
Two words: targeted advertising.
The first to do this will win - it’s classic chicken and egg. Bloggers will only bother to register their content with the paper that has the most readers for them, and people will only subscribe to papers that can provide them with their preferred content.

In reality though, the top newspapers will never do this. They mistakenly believe their strength is in breaking news and in-depth reporting of current events, not in providing low-cost, advertising-supported, globally-distributed printouts.

Sadly, it’d be a lot harder for a startup like Instapaper to do this - you really need a way to get print delivered to my door by 8am every morning for the price of a coffee and you need a brand that will convince bloggers to share their content with you in exchange for some revenue and recognition. You can’t get there from a standing start.

So expect to see Instapaper focus on iPad and iPhone and Kindle, and for the insanely massive print market to continue to be ignored by everybody who could make it work.

It’s a pity. For a moment there, I thought I was looking at the future.

I had fun writing a little board game recommendation app last...

Sat, 09 Oct 2010 05:28:00 -0400

I had fun writing a little board game recommendation app last week. Well, the last two weeks I guess. Think Amazon recommendations meets Google Instant. Here it is!

Note: If anyone’s interested, I’ll follow up with a “Making of…” post next week sometime. I’ve always wondered what people actually do when they write these ‘weekend’ apps. If you’ve any questions, ask away and I’ll try to answer them in the post.

What Do Bingo Card Creator And Google Have In Common?

Tue, 21 Sep 2010 08:21:00 -0400

A serious point underlies the flippancy in Single- vs Co-Founder: It’s Like Star Wars - we use the word ‘startup’ to refer to a wide range of different businesses, yet treat them as if they were basically the same thing. Advice for one doesn’t necessarily apply to the other, so we should ask: what do we mean when we talk about a startup?

Not so long ago, the definition of a startup was very literal:

A company that is in the first stage of its operations

During the dot-com booms, easily-accessible VC funding made the term virtually synonymous with the Silicon Valley, high-investment model. Until recently I’d always heard of it in this context, but I’ve never seen it clearly defined in these terms. Today there are a wide range of businesses with completely different operating models being referred to as startups; just take a look at these:

Bingo Card Creator - Patrick is the poster-boy for single-founder, organically-grown freedom-based software businesses. He started BCC as a side-project and has grown it to a small business empire that he runs as sole founder and owner. It’s not clear how it could have benefitted from VC funding or a co-founder. Is BCC a startup?
DuckDuckGo - Gabriel Weinberg is himself an angel investor, so does the plucky alternative search engine count as having taken investment already? As far as I’m aware, Gabriel works on DDG alone, but I suspect he wouldn’t rule out taking VC funding should its growth explode. Is DDG a startup?
Apple - Woz and Jobs didn’t take any external pre-IPO funding, yet their garage-born disruptive technology business is one of the classic startup stories. Would Apple be a startup in today’s terms?
Google - The archetypical Silicon Valley story. Two people with great ideas and technology. $100k angel investment before they even incorporated. $25m from Sequoia and Kleiner Perkins. $1.67b IPO. Most famous startup of our time.

How can we talk about what a ‘startup’ needs without differentiating between these kinds of companies? Should we talk about ‘funded startups’ and ‘organic startups’? Are there two (or more) discrete categories, or just points along a continuum?

Clearly we can imagine a continuum along the scale of funding / growth. Personally, I suspect profitability is greatest at the ends with a big dip of death in the middle. Have you ever heard of a company who said:

Now we’ve secured our series A round we’re planning to grow a bit faster than we can afford to, but not exponentially. Just somewhere in the middle.

Maybe I’m wrong and there are successful strategies like this, but it’s non-obvious. If there are discrete categories, what defines them?

Growth vs. Revenue - are they using external funding to grow faster than their natural rate?
Potential for Disruption - are they trying to redefine a market, or just find a profitable niche?
Exit Strategy - IPO or profitability with dividends?

It’s interesting to see how historical cases are divided by these classifications; as a startup or whichever kind it’s clearly going to be more helpful to follow the example of companies who match your profile, but which are the classifications that count?

I don’t have any answers, I just have questions for you and everyone else in the community:

What sort of company do you mean when you say ‘startup’?
Is there a continuous scale between Bingo Card Creator and Google, or are they discrete types of startup?
How would you classify startups into groups, such that advice for one startup is likely to apply to others in its group? Is this even possible?

Answering these, even just trying to, will make future discussions more transparent and better-defined. How do you define and categorize startups?

Single- vs Co-Founder: It's Like Star Wars

Mon, 20 Sep 2010 10:43:00 -0400

Does a startup benefit from two or three co-founders? Can single founders compete? Every so often this topic bubbles up into my ~~consciousness~~ rss feed. Lots of single founders say “It works fine for me” and lots of co-founders say “We probably wouldn’t have made it alone”.

These discussions are often had completely at cross-purposes: people persist in talking about very different kinds of companies. Look, it’s like Star Wars:

It’s Like Star Wars

Senator Palpatine has a dream. He’s seen this neat hack he can use to disrupt the established order and build a new empire, in which all the money (and power) goes to him. He needs to move fast, grow exponentially, work continuously. He needs a co-founder to help him in this big, high-stakes gamble against the world, so he goes out and gets one.

Han Solo also has a dream; the dream of personal freedom, of being his own man. He couldn’t be less interested in changing the world. He doesn’t want any part of the rebel alliance’s plans. He’s small fry, but he’s got what he wants - the ship he’s so proud of, his crew and the freedom to go and do what he chooses, beholden to no-one [1] - a lifestyle business.

See What I Did There?

The classic venture-funded startup is a high-pressure, all-or-nothing gamble against the world; in this environment it’s easy to see why the advantages of co-founders matter so much - doubly so from PG’s perspective as an investor in such companies. Anyone investing in Palpatine [2] without Anakin would have lost their money (as it is, all the smart investors cashed out long before Return of the Jedi, right?)

Most single-founders I’ve spoken to personally or read regularly are actually building lifestyle businesses. They work for their freedom, their independence, the ability to choose their own destinies. Sure, they’d love to make it big, but they’re happy to take it slow. How would you find a good co-founder for a lifestyle business? You’ve got two different lives to lead and the whole point of the business is getting to do it your way. In these circumstances, a single founder with employees or contractors makes a lot of sense. They’re Han Solo, in their very own Millenium Falcon, and they like it.

So next time you read, write or comment on an article comparing the merits of single and co-founders; next time you feel your personal choice undermined by someone else’s argument, stop and ask yourself: is this about forging an empire, or being master of your own destiny? They’re not the same thing. [3]

[1] Well, except his debtors, although that’s nothing you can’t cure by shooting first. And yes, yes, Han-Solo had his long-time friend as first-mate, but he was already on his entreprenurial path before they got together and he’s clearly calling the shots. Don’t go all Chewbacca Defense on me here

[2] I don’t want this negative image to distract - I’m not trying to compare Silicon Valley to the Sith Lords here, it’s just for illustration. Although now I think about it, they do have my data by the throat.

[3] This is really true. Palpatine has power and resources Han Solo can only dream of, but Han Solo has freedoms that the emperor can never have. Great responsibility and all that. Also, Han Solo doesn’t get stabbed in the back by an ungrateful co-founder after they make it big, but this is getting off-point real fast…

Unexpected Good News

Thu, 16 Sep 2010 05:34:00 -0400

Configuring ‘expect’ on linux today:

checking for stdint.h… yes

checking for unistd.h… yes

checking if the compiler understands -pipe… yes

checking whether byte ordering is bigendian… yes

…

checking for sin… no

Well, that’s a bit of good news ;-) I wasn’t aware I’d granted OAuth credentials for my eternal soul, though. What’s the expiry on those?

A Time For All Things

Wed, 08 Sep 2010 22:35:00 -0400

A while ago Jason Sage suggested the wisdom and beauty of this poem apply quite naturally to programming. He was very, very right - so with apologies to the unknown original author:

3ccl35145735: 3

1 There is a time for all things;
a season for every activity under heaven:

2 a time to hack it together and a time to refactor the mess;
a time to be clever and a time to be featured on thedailywtf,

3 a time to throw descriptive exceptions and a time to return false;
a time to release early and a time to refrain from releasing at all,

4 a time to think about the problem and a time to ask stackoverflow;
a time to copy from a random forum post and a time to wish you hadn’t,

5 a time for getting on with work and a time for reddit
and for hacker news and dilbert and xkcd and penny arcade,

6 a time to GET and a time to POST;
a time to sanitize the database inputs and a time to sanitize the database inputs properly,

7 a time to make the method public and a time to make it private again;
a time for dynamic typing and a time for runtime errors on the production server,

8 a time for the strategy pattern and a time for switch {};
a time to argue about it and a time to regret not arguing enough,

9 What does the programmer gain from his toil? I have seen the burden God has laid on men.

10 He has made everything beautiful in its time,

10a Except C++

10b And PHP

10c Don’t get me started on actionscript, OK?

10d Fucking actionscript

11 He has set eternity in the hearts of men, yet they cannot fathom what He has done from beginning to end,

11a nor indeed what they themselves were doing yesterday when they tried to implement memoized tail-recursion using templates.

12 Everything God does will endure forever, but with any luck my code will be deprecated in the next release.

12a Or at the very latest the one after that.

12b Oh please God don’t make me maintain this for the next twenty years.