How a stale-check race in DynamoDB's DNS layer took down AWS US-EAST-1 for 14 hours

On Monday, October 20, 2025 at 06:48 UTC, DynamoDB's regional endpoint (dynamodb.us-east-1.amazonaws.com) started returning an empty DNS record. Every IP address was gone. The outage lasted roughly 14 hours, hit 140 AWS services, and took down Slack, Snapchat, Ring, and Coinbase.

The root cause was a Time-of-Check-Time-of-Use race condition in DynamoDB's internal DNS management system. A stale DNS plan, applied late by a lagging component, overwrote a newer plan, and a cleanup process with no guard against deleting the active state wiped the record entirely.

DynamoDB is a foundational layer-one service, so its 3-hour outage cascaded into EC2's physical server management and Network Load Balancers. DNS was back by 09:40 UTC, but the regional damage was just getting started.

DynamoDB manages its own DNS via a DNS Planner that monitors load balancers and generates plans, plus three independent DNS Enactors (one per AZ: us-east-1a, 1b, 1c) that apply those plans to Route 53.

One DNS Enactor hit extreme latency, 10x to 100x normal, while applying an older plan. Meanwhile the Planner generated newer plans, a second Enactor applied them, and triggered a cleanup to delete old plans. The slow Enactor finally committed its stale plan, overwriting the newer state. The cleanup process, lacking any guard against deleting active records, then deleted that plan too.

Manual operator intervention took over 2 hours to fix the DNS empty state. The system had no path to self-heal because every component believed it had done the right thing.

EC2 uses the Droplet Workflow Manager (DWFM) to manage physical servers, and DWFM stores lease heartbeats in DynamoDB. During the 3-hour DynamoDB outage, DWFM could not complete state checks, and broken leases spiked from roughly 10^2 to 10^5.

When DynamoDB returned, DWFM tried to re-establish all leases simultaneously. The queue grew so long that work timed out before completion, entering congestive collapse. Engineers had to restart DWFM servers to drop in-memory queues; EC2 launches stayed impaired until 20:50 UTC.

The launch backlog also delayed Network Manager's state propagation, so new instances came up without network connectivity. NLB health checks failed on these isolated instances, marked them unhealthy, then passed them later once networking caught up, causing constant capacity flapping.

AWS has disabled the DynamoDB DNS Planner and DNS Enactor automation worldwide until the race condition is fixed. The cleanup that deleted the active DNS record had no guard against removing the current state, and that gap turned a stale write into a regional outage.

On the EC2 side, AWS plans velocity controls for NLBs to limit capacity removal during AZ failovers, and queue-size-based rate-limiting to prevent future congestive collapse.

The broader pattern is metastable failure. The DNS race was the trigger, but the 14-hour duration came from downstream systems that entered self-reinforcing bad states and could not climb out without human intervention.