Yesterday we had a bad outage. From 22:25 to 22:58 most of our servers were down and serving 503 errors. As is common with these scenarios the cause was cascading failures which we go into detail below. Every day we serve millions of API requests, and thousands of businesses depend on us - we deeply regret downtime of any nature, but it's also an opportunity for us to learn and make sure it doesn't happen in the future. Below is yesterday's postmortem. We've taken several steps to remove singl
Wow. 2015 was quite the year for Clearbit! It's tricky fitting everything we've done into a single blog post, considering last year encompassed most of our young company's life, but here goes. In 2015 we went from a team of two, to seven. From 50 customers to 500. From processing a few hundred API requests a day to millions. We now power the business intelligence data behind much of the valley, and this year we're setting our sights much higher. I'm very proud of how far we've come in a year,
Startup stock options often [http://zachholman.com/posts/fuck-your-90-day-exercise-window/] get a bad [http://jvns.ca/blog/2015/12/30/do-the-math-on-your-stock-options/] rap [https://news.ycombinator.com/item?id=10811570], and with good reason. Employees who've given years to a company can end up with nothing — or have to fork over huge amounts of money and suffer a large tax liability. Quite honestly this sucks! To be fair, the system we have in place isn't necessarily the fault of companies a
In Part 1 [https://clearbit-blog.ghost.io/blog/servers-part-one] of this series we gave you an overview of the deployment process at Clearbit. In this post I'm going to talk about our how we use fleet to run our services. > Note: We use Amazon's EC2 system. In this article when I refer to a machine I'm really talking about an EC2 instance. I've used machine to avoid confusion with an instance (runnning process) of an application. Life before Fleet Before migrating over to CoreOS/Fleet we were
Engaging stories and exclusive data, designed for our best customers. One useful issue each month.