Postmortem of yesterday's downtime

Companyby Alex MacCaw on June 10, 2016

Yesterday we had a bad outage. From 22:25 to 22:58 most of our servers were down and serving 503 errors. As is common with these scenarios the cause was cascading failures which we go into detail below. Every day we serve millions of API requests, and thousands of businesses depend on us - we deeply regret downtime of any nature, but it's also an opportunity for us to learn and make sure it doesn't happen in the future. Below is yesterday's postmortem. We've taken several steps to remove singl

Streak's InboxSDK

Companyby Alex MacCaw on February 17, 2016

Building on top of Gmail is hard. Like really hard. Google doesn't provide any official JavaScript APIs, and indeed seems pretty laissez faire about people building on top of the platform in general. Then there's the matter of the generated source code. Just go and click 'View Source' on Gmail. If your browser doesn't end up crashing, you'll be unpleasantly surprised with a spaghetti of generated markdown The rewards for integrating with Gmail though, are large. It's where a lot of people spen

2015 year in review

Companyby Alex MacCaw on January 19, 2016

Wow. 2015 was quite the year for Clearbit! It's tricky fitting everything we've done into a single blog post, considering last year encompassed most of our young company's life, but here goes. In 2015 we went from a team of two, to seven. From 50 customers to 500. From processing a few hundred API requests a day to millions. We now power the business intelligence data behind much of the valley, and this year we're setting our sights much higher. I'm very proud of how far we've come in a year,

Our Stock Option pledge

Companyby Alex MacCaw on January 12, 2016

Startup stock options often [] get a bad [] rap [], and with good reason. Employees who've given years to a company can end up with nothing — or have to fork over huge amounts of money and suffer a large tax liability. Quite honestly this sucks! To be fair, the system we have in place isn't necessarily the fault of companies a

Docker & CoreOS - Part 2

Engineeringby Rob Holland on September 12, 2015

In Part 1 [] of this series we gave you an overview of the deployment process at Clearbit. In this post I'm going to talk about our how we use fleet to run our services. > Note: We use Amazon's EC2 system. In this article when I refer to a machine I'm really talking about an EC2 instance. I've used machine to avoid confusion with an instance (runnning process) of an application. Life before Fleet Before migrating over to CoreOS/Fleet we were

Join our newsletter

Engaging stories and exclusive data, designed for our best customers. One useful issue each month.