Why Slack Rebuilt Around Cells After Its Cache Collapsed

On the morning of Feb 22, 2022, a routine rollout upgrading the Consul agent across the fleet, 25% at a time, quietly lit the fuse. When an agent restarts on a memcached node, that node briefly deregisters from the service catalog. Mcrib, Slack's control plane for the cache tier, saw those nodes vanish, promoted spares, and flushed the ones that rejoined to avoid serving stale data. The cache hit rate started sliding.

It hit at peak daily traffic. Loading a user's Group DM membership on boot reads from a Vitess keyspace sharded by user, not by channel, so any cache miss became a scatter query touching every shard. With the cache gutted, nearly every boot fired a full-keyspace scatter, database load grew faster than the miss rate, queries timed out, and the cache could never refill. That is a textbook metastable failure: the system stays stuck in overload even after the trigger is gone.

The way out was to aggressively throttle new boot requests, trading reconnecting users' access to protect the ones already inside, until the cache slowly recovered. This was one of a run of cascading failures Slack lived through in this era, and the redesign that reshaped the architecture came out of a different incident entirely.

Eight months earlier, on June 30, 2021, an AWS network link between one availability zone and others in us-east-1 began failing intermittently. Slack spreads its core compute across multiple AZs in that region precisely so one bad zone shouldn't matter, since a region's availability is supposed to beat any single AZ's. Users saw errors anyway.

The reason it leaked everywhere is structure. A single user API request fans out into hundreds of RPCs to backends, and all of them must succeed. Worse, services weren't isolated per zone: a service in one AZ routinely called memcache, pub/sub, search, and Vitess sitting in other AZs, a criss-crossing mesh. One degraded link could poison responses built anywhere in the region.

The error rate sat around 1%, out of SLO for a service targeting 99.99% availability, under an hour of downtime a year. Engineers staring at per-AZ graphs wanted a single lever to drain the bad zone and had none, and nobody trusts improvising one live: doing something for the first time during an incident is exactly when you don't want to.

The rebuild's core idea is siloing: a service only receives traffic from, and only sends traffic to, servers in its own AZ. Each service effectively becomes N virtual per-zone copies, so a fault stays trapped in the zone where it started instead of bleeding into responses region-wide.

Siloing gives you a clean boundary to act on, and Slack wrote four hard rules for acting: drain most traffic off an AZ within 5 minutes, cause zero user-visible errors while doing it, move in increments as fine as 1% so recovery can be tested cautiously, and, critically, the drain must not depend on the failing AZ's own resources. That last rule ruled out clever tricks like SSHing into every host to force a healthcheck failure, since the whole point is to work when the zone is already gone.

Slack had already spent years moving from HAProxy to Envoy at the edge, all configured by Rotor, its in-house xDS control plane. Draining reuses two Envoy features it already had: weighted clusters, and dynamic weights pushed via RTDS. To drain a zone, Rotor tells the edge load balancers to reweight that AZ's target clusters toward zero. In-flight requests finish; every new request routes to the other zones.

This clears all four goals at once. Propagation runs in seconds and Envoy applies weights immediately, drains are graceful with no abandoned requests, weighting supports 1% steps in both directions, and the edge load balancers live in different regions from us-east-1 behind a regionally replicated control plane, so the drain survives the very AZ failure it exists to fight.

None of this was a research breakthrough. Bethea's team is blunt that they "didn't even invent an algorithm"; the difficulty was executional and organizational, spread across a heterogeneous stack with services in Hack, Go, Java, and C++. An earlier attempt at the same migration had failed to get off the ground before the successful one landed.

Isolation isn't free. Siloing means every service runs redundantly in each zone, duplicated compute and duplicated caches, and all the drain logic concentrates in the Envoy and Rotor layer. That layer becomes the most load-bearing part of the system, and it sits outside the isolation it hands to everyone else.

Slack also deliberately chose human judgment over automation. Detecting a gray failure automatically is genuinely hard, so rather than trust auto-remediation, an operator watches the graphs and taps the drain button. The lever is fast and reversible, and a person decides when to pull it.

Data is the part traffic-draining doesn't fully solve. Vitess is strongly consistent, so each shard has a single writable primary, and if that primary sits in a drained zone it still has to be failed out separately, a harder open problem. The migration covered Slack's most critical user-facing services over roughly 1.5 years.