Twitter Tales

These articles form a series of memorable moments from working at Twitter in its early days. As told by Brady Catherman, Twitter employee 51, and an early Operations Engineer that worked on all sorts of crazy and funny experiences during his three years working there.

During Twitter's early days, when the company was still less than 100 engineers, a small computer became a crucial piece of infrastructure. This is the true story of Twitter's infamous "Load Bearing Mac-Mini."

An errant feature launch managed to erase one poor unsuspecting user from Twitter. This user got stuck in a broken state, able to tweet and see a timeline, but nobody could see their profile. This article explores the reason why it happened and provides tips for preventing this type of failure on your own site.

The 2010 World Cup was a pivotal moment in Twitter's history. It both established the model for incident management, as well as a process by which debugging could be done on exceptionally large distributed systems. It started with a total failure, but ended with a team coming together and finding the convoluted cause just in time.

In 2010 Twitter embarked on a project to move from managed hosting into a new, bespoke data center which all went completely wrong in no time flat. This is a quick write up of some of the catastrophic issues we encountered along the way.

Twitter used to have a system to help teach people to lock their work laptops. This is a story of how that system was used to get real brownies in the office.

Restarting Memcached at Twitter was always an extremely problematic experience. We often took great lengths to avoid restarting them at all, and when that wasn't possible we often had to jump through hoops to keep things working. This article explores some of those events and the eventual fixes that eliminated the problems.

In 2010 Twitter embarked on am ambitious project to replace its MySQL tweet store with Cassandra, a newfangled distributed Key Value store. Called project `Snow Goose` this ended up leading to a crazy hack of a project called Project Dirt Goose.