System Design · Issue 15
Slack's 2022 cache-miss cascade: how one missing key fanned out across every shard
A routine rollout swapped out cache nodes, and a single miss turned into a query to every database shard.
The setup
A cache layer hiding an expensive query
Slack serves millions of concurrent connections through a real-time messaging backend built on Memcached and Vitess. A Memcached tier, scaled horizontally by Mcrouter with consistent hashing, sits in front of the databases. A control-plane component called Mcrib watches Consul for service discovery and swaps in a spare instance whenever a cache node becomes unavailable.
Cache hit
one node, sub-ms
Cache miss
scatter to every shard
The cache hides a latent problem. Slack uses Vitess to shard MySQL by user ID, but GDM membership queries are keyed by channel ID. With no routing key, Vitess must issue a scatter query to every shard in the keyspace. A long-TTL Memcached entry normally makes this invisible.
The mechanism
One missing key becomes N queries
When a GDM channel's membership entry is absent from cache, the application sends a query to Vitess carrying only a channel ID. The keyspace is sharded by user ID, and the query has no user ID. Vitess cannot route it to a single shard, so it fans the query out to every shard and merges the partial results.
Cache miss (channel ID)
- Shard 1
- Shard 2
- Shard 3
- Shard N
A single missing key produces one query per shard. At Slack's scale, the GDM keyspace has enough shards that each miss is a real fan-out. Under normal conditions the long-TTL cache makes misses rare, so the cost stays hidden. But the cost per miss scales with the number of shards, not with the number of users in the channel.
The cascade
A rollout that emptied the cache
On February 22, 2022, Slack was rolling out updated Consul agents in percentage-based steps. Two 25% steps had succeeded the prior week. The third step restarted agents on memcached nodes, causing each to deregister from the service catalog so Mcrib replaced it with an empty spare.
As empty spares rotated in, the cache hit rate dropped. Each miss sent a scatter query to every shard, and database read load grew superlinearly. Queries timed out before returning results to populate the cache, so the next request for the same channel also missed. Lower hit rates caused higher database load, and that load prevented the cache from refilling.
The tradeoff
Throttling users to save the system
The most affected operation was the client boot process, where a client fetches its initial workspace state. Responders applied a restrictive throttle to boot requests: unbooted users could not connect, but already-booted users retained normal service.
Reducing load reduced database timeouts, which let the cache finally refill.
What is a metastable state?
A metastable state is one where the system sustains itself under normal load but cannot recover from a perturbation without external intervention. Slack's system was stable with a high cache hit rate. The Consul rollout was the perturbation that pushed it past a tipping point. Once in the overloaded state, the feedback loop between cache misses and database timeouts kept it there. No amount of waiting would fix it; the load had to be reduced externally to let the cache refill, breaking the cycle.
- Throttle boots
- DB timeouts drop
- Cache refills
- Raise limit slowly
- Recovered
Recovery was not smooth. When responders first raised the boot rate limit, they raised it too far, database load spiked, and errors returned. They had to cut the limit back and increase it in small increments to stay below the threshold where the cascade would retrigger. Blocking new users is painful for a communication product, but fewer boot requests meant fewer scatter queries, fewer timeouts, and eventually a cache that could refill on its own.
The lesson
Every scatter query behind a cache is a latent cascade
Why it matters
At Slack, a single cached key per GDM channel was the only thing preventing a membership query from touching every shard. The cache was load masking: it hid a scatter query whose cost scaled with shard count. When service-discovery churn emptied cache nodes, the database saw that cost all at once, and the overload prevented the cache from refilling. Any scatter query behind a cache is a latent cascade, invisible until something drops the hit rate.
The short version
- A Consul agent rollout replaced memcached nodes with empty spares, dropping the cache hit rate.
- A single cache miss for a GDM channel triggered a scatter query to every Vitess shard.
- Database timeouts prevented the cache from refilling, locking the system in a metastable feedback loop.
- Responders broke the cycle by throttling client boots, trading connectivity for cache recovery.
- Recovery required small, incremental throttle increases to avoid retriggering the cascade.