Deep Dive · Issue 18

The stale-state overwrite: why a one-time freshness check is a time bomb in distributed coordination

A lock you hold is not a lock you own. If your process pauses, the lease expires, and a freshness check can't save you.

16 September 2026 A paused process holding an expired lock while another client approaches the same resource

The problem

Your lock expired while you were asleep

A paused process holding an expired lock while another client approaches the same resource

Martin Kleppmann diagnosed this in 2016: a client acquires a lock, reads shared storage, modifies it, writes back. If a GC stop-the-world pause exceeds the lease TTL, the client wakes up after the lock expired and another client re-acquired it, then overwrites newer state. HBase had this bug.

The canonical failure timeline for lease-based locking. Two clients end up in the critical section, and the later write loses.

The failure is inherent to lease-based locking, not a bug in any implementation. A one-time freshness check ("is the lock still mine?") does not help because the pause can occur between the check and the write. CWE-367 names this: a time-of-check to time-of-use race, where the gap between checking and acting is the window for corruption.

The mechanism

Fencing tokens: the storage layer decides

A storage gatekeeper rejecting a lower-numbered token and accepting a higher one

The fix is a fencing token: a monotonically increasing number assigned by the lock service on each acquisition. The client includes the token with every write in the critical section. The storage layer tracks the highest token it has seen and rejects any write with a lower number.

Fencing token — A monotonically increasing number assigned by the lock service on each lock acquisition, included with every storage write, and enforced by the storage server to reject stale writes.

Client 1 paused with token 33. Client 2 wrote with token 34. When Client 1 wakes, the storage layer rejects its stale token.

Three components are required: the lock service generates strictly increasing tokens, the client propagates the token with every operation, and the storage layer enforces the ordering. Drop any one and the protection vanishes. The lock service generates the token; the storage server enforces it. Both halves are required.

The numbers

Which lock services can actually fence

Three lock implementations compared: DynamoDB, Redlock, and ZooKeeper

Redlock: clocks are the fatal flaw

Redlock distributes locks across 5 independent Redis instances and requires a quorum of 3. Its correctness depends on clocks advancing at the same rate. If one instance's clock drifts ahead from NTP or VM migration, it expires a lock early while others still consider it valid, and two clients hold the lock simultaneously.

Clock skew attack on Redlock. R3 expires early because its clock is ahead, letting Client B form a different quorum while Client A still believes it holds the lock.

ZooKeeper: natural tokens, but no free lunch

ZooKeeper avoids clock dependency with session-based expiration instead of TTL timers. A sequential ephemeral node's number (or zxid) is a natural monotonic fencing token, globally ordered across the cluster. But "built-in fencing" is a myth: the storage layer must still enforce tokens. If the database does not reject stale ones, you are no safer than with Redis.

The tradeoff

Atomic check-and-act, or architecture

A conditional UPDATE rejecting 19 concurrent requests and accepting one

A conditional UPDATE sidesteps the fencing problem entirely by making the check and the write atomic. In a reproducible lab with 20 concurrent requests and a 1-voucher limit, no locking leaked 16 of 20; a naive non-atomic Redis lock leaked 9 of 20 over HTTP/2. A conditional UPDATE held at exactly 1 of 20.

UPDATE daily_quota
SET granted = granted + 1
WHERE member_id = :member_id
  AND granted < :limit;

Conditional UPDATE. The WHERE clause re-evaluates against the latest committed row, so the check and the write happen in one atomic step.

CWE-367 classifies this as a TOCTOU race, and the conditional UPDATE closes the gap because in READ COMMITTED (Postgres's default), the UPDATE re-reads the latest committed row at execution time and re-evaluates its WHERE clause in one atomic step. But the lab proves DB-internal atomicity only. When side effects live outside the database (external payment calls, coupon cancellations, point grants), you need idempotency keys, the outbox pattern, or compensating transactions.

Closing

The check is never the last word

Why it matters

The stale-state overwrite is the default failure mode of any system that checks freshness once and then acts later. GC pauses, network delays, and clock drift are not exceptional in distributed systems; they are the environment. The systems that survive are the ones where the storage layer, not the client, has the final say on whether a write is stale, and where the lock service and the storage server are treated as two halves of one mechanism.

The short version

A one-time freshness check cannot prevent stale writes because the process can pause between the check and the write.
**Fencing tokens** fix this: the lock service generates a monotonic token, and the storage layer rejects any write with a lower token than it has already seen.
**Redlock** depends on bounded clock drift and cannot generate fencing tokens; **ZooKeeper's** sequential nodes can, but only if the storage layer enforces them.
A **conditional UPDATE** makes check and write atomic inside the database, but external side effects need idempotency keys, outbox, or compensating transactions.