The Cache Stampede That Ate Slack: Inside the 2022 Cascading Failure

On the morning Slack's own writeup nicknamed "2-22-22," the service fell over just after 6 a.m. Pacific. The thing that set it off was a maintenance task so routine it had already run twice the week before without anyone noticing.

Nothing "broke" in the usual sense. Three healthy systems, each behaving exactly as designed, interacted at peak traffic and tipped the whole datastore tier into a self-reinforcing overload. It ended up being one of three major Slack outages that year.

Slack stores data in Vitess, a system for sharding MySQL across many machines, organized into keyspaces that are each split by a chosen key. The keyspace at the center of this story holds channel membership, sharded by user.

In front of Vitess sits a memcached tier. Mcrouter spreads cache reads across the fleet by consistent hashing, and Mcrib is the control plane that keeps Mcrouter's host list current, watching Consul and swapping in spare cache instances whenever a node drops out. Every piece here is doing sensible, careful work.

Slack was mid-rollout of a Consul agent upgrade, applied to 25% of the fleet at a time with slow, sequential restarts. Two earlier 25% steps had gone by cleanly, so there was no reason to expect this one to behave differently.

When a Consul agent restarts, it briefly deregisters and re-registers the node's services. Mcrib reads that as the memcached node going away and promotes a fresh, empty spare to cover for it, flushing rejoining nodes so no stale data survives. Sensible on its own, but as nodes cycled through this at peak traffic, a real chunk of the cache went cold at once.

When a Slack client boots, it fetches the membership of your Group DM lists. That data lives in the user-sharded keyspace, which is great for "what channels is this user in," but answering "who is in this GDM" means asking every shard, a pattern Vitess calls a scatter query.

Because GDM membership never changes, it caches with a long TTL and is a near-guaranteed hit, so the scatter query almost never actually runs. With cache nodes cycling out, a single missing channel forced a full scatter across every shard. Read load climbed superlinearly with the miss rate until most queries against those tables timed out, which meant the cache could never refill. The failure now fed itself.

The slow, failing boot requests exhausted resources in the database tier and started blocking users whose clients had already loaded fine. Slack's mitigation was to throttle client boot requests, a deliberate trade that sacrificed reconnecting users to protect the ones already inside.

The first throttle helped, so engineers raised the boot limit again, overshot, and watched load spike right back before dialing it down and creeping up in smaller increments. This is what the postmortem calls a metastable state: once a system tips into cascading overload, it stays there until something external cuts load or adds capacity.