System Design · Issue 19

Why AWS Chooses to Fail 5% of Customers Completely

AWS would rather fail a small slice of customers completely than degrade for everyone. Here's the architecture behind that choice.

19 September 2026 A failure blast radius spreading across all availability zones and regions at once

The problem

Failures that travel with code

A failure blast radius spreading across all availability zones and regions at once

Multi-AZ and Multi-Region replication are the standard answers to hardware failure. They do nothing for failures that travel with the code and data. A bad deployment, a poison-pill request, a configuration change applied everywhere, or a single tenant hammering a shared resource arrives identically in every AZ and every Region at the same instant.

A bad deployment arrives identically in every AZ and Region. Multi-AZ and Multi-Region do not contain it.

AWS has used cell-based architecture internally for more than a decade to solve exactly this class of failure. The Well-Architected guidance frames the design question directly: is it better for 100% of customers to experience a 5% failure rate, or for 5% of customers to experience a 100% failure rate? Cell-based architecture is what you build when the second answer is correct.

Is it better for 100% of customers to see a 5% failure rate, or 5% to see 100%?

The mechanism

Three components, one thin router

Cell router directing traffic to independent cells, with control plane off the request path

A cell-based system has exactly three parts. The cell router directs each request to the right cell based on a partition key like customer ID. It must be the thinnest possible layer, simpler and more reliable than anything it fronts. Behind it sit cells, each a complete independent workload instance with its own compute, storage, and state, sharing nothing.

The router is the thinnest possible layer. The control plane is off the request path and only provisions, migrates, and deploys.

The control plane provisions cells, migrates customers, and deploys code, but stays off the request path. Control planes fail more often than data planes, so the router loads its cell mapping into memory from S3 and serves from that copy indefinitely. Even if the control plane, S3, or an entire zone is down, the router still directs traffic. Stale routing beats no routing.

Static stability — A data plane that keeps serving from cached in-memory state when its control plane is down. The router treats its mapping as a snapshot, not a live dependency.

The numbers

Blast radius shrinks linearly

Ten cells with one failed, showing 90% unaffected

Cells have a fixed maximum size. You cap each component and scale by adding cells, not by growing them. The cost concern is often misplaced: an application with 30 hosts might keep the same 30 hosts, just grouped behind a router.

\text{blast radius} = \frac{1}{N_\text{cells}} \quad \Rightarrow \quad \text{unaffected} = \frac{N-1}{N}

With 10 cells, one failure leaves 90% unaffected. With 3, it leaves 67%.

Spread a workload across 10 cells and a failure in one leaves 90% of requests unaffected. Across 3 cells, the worst-case initial failure is 33% of capacity instead of 100%. Recovering a database with 0.1% of the data is also far faster than recovering 100%.

Deployment as blast-radius control

Deployments proceed cell by cell in waves, with the first cell acting as a canary. Bake time between waves ranges from 15 minutes to several hours. If a bad change surfaces, it hits one cell, not the fleet.

The tradeoff

What cells don't contain

The cell router as a shared dependency that cannot be cellularized

Cells only contain what is actually inside them. The cell router itself is a shared dependency that cannot be cellularized, and it must be operated at a higher reliability bar than everything it fronts. If the router fails, every cell is unreachable. Cells also provide no protection against any other shared dependency that all cells rely on.

The router is a shared dependency excluded from the isolation guarantee. It must be more reliable than anything it fronts.

The operational cost is real. Certificate rotation, patching, capacity planning, and on-call runbooks all scale with the number of cells. Data that naturally crosses cell boundaries, like account settings, billing, and notifications, creates severe consistency complexity. Cells are explicitly not a disaster recovery mechanism.

Which failure classes cells actually contain

Multi-AZ handles host and rack failures. Multi-Region adds protection against AZ and Region impairment. Cells add the layer nothing else covers: bad deployments, poison-pill requests, tenant overload, and operational mistakes on the live system, all scoped to a single cell. The one class cells cannot touch is a failure of a dependency shared by every cell, including the router itself.

Wrap-up

Containment over redundancy

Why it matters

The cell-based pattern reflects a specific bet: that the worst failure mode in a large distributed system is not hardware loss but correlated software failure, where the same bad code or data hits every replica simultaneously. Redundancy across AZs and Regions does not help when the failure arrives with the deployment. Cells accept a small permanent blast radius in exchange for never having a large one, and they push the hard reliability requirement onto a single thin router rather than spreading it across the fleet.

The short version

Cells contain failures that travel with code and data: bad deploys, poison-pill requests, and tenant overload, which Multi-AZ and Multi-Region cannot stop.
A cell-based system has three parts: a thin **cell router**, independent **cells** that share no state, and a **control plane** kept off the request path.
The router loads its mapping into memory and serves from the cached copy indefinitely. **Stale routing beats no routing.**
Blast radius shrinks linearly with cell count: 10 cells means one failure leaves 90% unaffected.
The cell router is a shared dependency that cannot be cellularized, and cells are not a DR mechanism.