Adam Smith's Blog

Investing A Million Bucks Into Performance and Stability

(This blog entry is based on a long comment I posted recently on Hacker News.)

Last week Xobni announced that we’ve added the Blackberry Fund to our series B financing.

We also released a new version of our software alongside that announcement. The release got less attention because there aren’t any user visible features.

Truth be told, though, we invested about a million dollars into that release! And the investment will be critical to the success of Xobni.

So how is it possible to invest so much, on something so important, without adding new features?

If you’ve read the title of this post then you already know the answer. Other than consuming ungodly amounts of Reese's Peanut Butter Cups, we worked on two things -- performance and stability.

Outlook is a hostile environment for a product like Xobni. It has several different APIs, each with different quirky interfaces, side effects, and threading models.

There's no way around having complex APIs for Outlook. They expose programmatic access to the most complicated email application ever built. Just like Excel and MS Word, it's really hard to underestimate how feature rich this program is.

As evidence, I’d suggest you check out this unusual peak into the depths of Outlook's complexity.

Scroll down to the section titled "Individual bugs that are fixed." Wow!

Let's run through my favorite example of how complex these APIs are.

Imagine you have an ID for a message, and you want to open a draft reply to that message so the user can type in their reply and press send. It should work just as if the user hit the Reply button, or pressed Ctrl-R.

Easy right? That's what I thought, too...

The first API I tried seemed to work, but when the user pressed send the icon for the original message didn't change to the purple arrow to indicate that it had been replied to.

I found a second approach that didn't have that bug, except it turned out to save the draft of the message (if it is being composed for more than five minutes, or the user explicitly presses Save) STRAIGHT INTO THE INBOX, instead of the Drafts folder. It looked funky -- no sender name, no sent time, etc. Ouch.

I found another API to use. It set the right icon -- good -- and seemed to save drafts into the Drafts folder. Double check. Unfortunately as soon as the user started typing it showed up in Times New Roman, 12pt, as the default font. Doh.

One of these three approaches also wouldn't pre-populate the user's email signature.

Fourth API was a charm!

…not to mention that the ID for a message is allowed to change under certain circumstances, such as when it gets moved to a different folder.

I could really talk about the complexity of Outlook and the scenarios we've ran into for days. Below are some example bugs I pulled from the Outlook feb-09 cumulative update document linked above. Each of these can hide weird race conditions, thread starvations, or just plain old corner cases that only show up when the moon is in the seventh house.

  * Inefficient processing occurs in a loop during intermittent network connectivity.

  * If the store providers are disconnected early, the Outlook.exe process becomes unresponsive for a very long time.

   * When you right-click an item, the whole item is loaded into memory more frequently than necessary.

   * Unnecessary disk reads are performed for every time that a custom form icon is rendered.

One of the fun side effects of doing this work is that you end up seeing all of the bugs you've come across in OTHER Outlook addins. I was using TechSmith's Snag-It the other day and smiled when I saw a draft it had opened save to the Inbox before I pressed send. They were using API #2. :-)

Not to mention that all of these other addins are accessing the same APIs, sometimes with "interesting interactions."

And on top of these challenges there are several users who have giant mailboxes. We've seen users with almost a million emails loaded into Outlook at the same time.

Usually these are the people with 12" laptops, 2 GBs of RAM, and 5400 RPM hard drives.

When this happens, all bets are off. Outlook takes a long time to load their twelve PST files. If an addin is trying to load its stuff at the same time you get heavy disk contention and "sequential" read throughputs plummet to 1 MB per second. And god help you if Outlook needs to "repair" any of these PSTs.

..the list goes on and on. We've been on a four month odyssey.

The sad truth is that there are still issues with our performance and stability. We fixed all of the reproducible issues but there still remain computers out there with all kind of weird registry permissions issues, combinations of other addins that can conflict with Xobni, and so on. It’s not the wild west out there, but bandits do pop up from time to time.

It's been both tough and fun. I love my team. We’ve hit some real high notes together.

I think we do it for our users. Not to be trite, but I think it’s true. Everyone on the engineering team feels a pinch of pain when users have problems. [1]

But, the other side of the coin is that we have a lot of fun with the code. There is some pretty awesome stuff going on under the covers. I'll give three quick examples..

1) Xobni's data store sits strictly underneath the sidebar code in the stack. It was originally built to support Xobni Analytics, our first product from 2006 that bombed. Fortunately we got to leverage the same data store when creating the Xobni sidebar. It's very cool. When someone is building software leveraging our backend, say the sidebar, or the Invite Your Friends feature, the code ends up looking like this:

   foreach(var mail in new MailIterator()) {
       Console.WriteLine(mail.Subject);
    }

This code will print 10k subjects per second, from disk! And it's from a key-value store, so it's easy to add new data fields and types.

Our data store is darn useful. Just two days ago I wrote some code against it to get some important data for a new project we’re working on. It looks like pseudocode!

2) Not only that, but the data store is built to be client agnostic above "layer 1" where we interface with the mail client. So when we wanted to integrate Yahoo Mail all we had to do was build the adapter piece that knew how to speak Yahoo's language, and suddenly the mail floats all the way up the stack and appears in the sidebar right next to Outlook emails. :-)

3) The areas where we display information from Facebook, LinkedIn, etc are all little embedded instances of Internet Explorer. The code for those extensions is all just HTML and Javascript. When the user changes the current email we invoke a specific JS function called updatePerson(), and there's a callback object the JS can use to make HTTP calls and write lines to the log file. This architecture, which was invented by someone smarter than me, allows us to pump these babies out quickly and without much QA risk to the other parts of the program.

It just doesn't get any cooler than this!

Anyway, back to the main subject of the blog post: performance and stability and the road ahead..

I think we’re all excited with what we've done, but mostly we're already starting to look look forward to the next generation of features for our customers. It's going to be an exciting rest-of-2009 ahead!

Stay tuned!

(Commercial: if you're a developer and are interested in being part of the team, send your resume to jobs@xobni.com!)

Controlled Fires and Source Control

I deleted a big chunk of code recently. It was written months ago and just wasn’t in use, so it needed to go. There’s always an emotional attachment to code, though, so deleting it is like burning up your old love letters. You know you should probably do it, but you’re conflicted and it’s painful.

I bet the people who set off controlled forest fires have a similar feeling. They have to burn down trees and fauna that they helped create. But you’ve got to toss out the old to make room for the new.

Luckily, though, there’s source control! You can delete whatever you want and it will still be in your revision history. You’ll never go back and restore deleted code, but knowing that you could helps you do what’s necessary.

As I write this, I remember that there’s a whole namespace in XobniCommon that needs to die. I’ll take care of that tomorrow.

Why programmers are bad at spin

Programmers survive by paying attention to the ugliness in their code. If you wake me up in the middle of the night and ask what I’m dreaming about, I’ll probably tell you what the three ugliest parts of our code are, and how I’m going to fix each of them. So when someone asks me about the code, my instinct is to describe the bugs. They’re just the first order of business! I think that’s why programmers have a hard time putting a positive spin on things, as compared to salespeople. We have a natural tendency to look for and focus on the things that are not perfect.

Reinventing Code

When it comes to ideas or products, you want to cover new territory. It's the same in the programming world. You want to build on others' abstractions. Sometimes, though, you just can’t help it. We’re writing our software on the .NET platform. In .NET, all user interface elements have their own window handle in the operating system. Window handles are expensive resources. Parts of our user interface needed a lot of user interface elements, but we couldn’t afford to allocate so many window handles. So I started writing what a critic would call a reimplementation of the System.Windows.Forms namespace [1], but using a windowless model. Whereas the .NET top level interface element is called Control, ours was called LightControl. We also had LightButton, LightPictureBox, etc. I was only implementing about 2% of the namespace, but I was still nervous. It should set off alarms whenever you’re doing something that is both low level, and similar to something that already exists. But I just couldn’t think of any alternative. This was a few months ago. I was relieved yesterday when I randomly stumbled upon a blog post by Raymond Chen in which he validates my strategy. Raymond Chen is the man. This isn’t the first time this has happened. I wrote our own file system about eight months ago because NTFS didn’t have the right performance characteristics. We needed to store lots of small files, but traditional file systems have horrid performance in this scenario. They seek the disk like crazy! We also didn’t need directories, shortcuts, or any operations other than { create, append, read entire file, delete }. Surprisingly, a good open source implementation didn’t exist, so we bit the bullet.


The Point First, don’t be afraid to code a variant of something that already exists. Do your homework first, but do what you’ve gotta do. Second, if you’re making something new, don’t hide power. You might be able to save someone from having to write new code. [2]
Notes [1] This would take a hundred developers two years to finish. Luckily we don't need all of the functionality, but more importantly we don't need an uncrashable API. [2] I'm not blaming MS on this one. Although many criticize them for not providing a windowless control framework, they couldn’t have done so just by exposing an extra option. They're not hiding power; it's just not there.