Paper · Issue 18
The 37/31 split: why configuration pushes cause nearly as many incidents as code pushes
Config gets a fraction of the safety tooling that code does, and it shows up in the incident stats.
The asymmetry
Config changes are at least as dangerous as code
When CS professor Vijay Chidambaram asked why so many outages involve configuration changes, Lorin Hochstein's answer was blunt: config changes are more dangerous than code changes. He rejected the idea that this is a base-rate illusion, arguing the effect is real rather than an artifact of selective attention or higher push frequency.
Code push
canary, stagger, rollback
Config push
straight to all nodes
The asymmetry is striking. Code pushes get staggered rollouts, canary deployments, automated rollback. Config pushes often go straight to production with none of those safeguards. The result is a class of changes at least as incident-prone as code, protected by a fraction of the tooling.
Config changes are more dangerous than code changes.
Four hypotheses
Why config doesn't get the code treatment
Hochstein offered several explanations. The first is that deployment tooling favors code. Salesforce's post-mortem noted code pipelines had automated stagger and canary requirements, but DNS config staggering was still being built. Netflix's Spinnaker supported regional rollout of code before config.
The second is that config is hard to test. Hochstein asks whether anyone has written a unit test for a configuration value. As a commenter put it, you can't test production configs without deploying them, because the difference between test and production is the configuration.
Two more forces
Config is also harder to stage because it is more likely to be inherently global than code. And in mature systems, operators change behavior through config rather than code, so the population of changes skews toward config while the safeguards skew toward code.
Is this just a base-rate effect?
Hochstein explicitly considered whether config changes simply happen more frequently than code changes, which would inflate their incident count without making any single change more dangerous. He rejected this explanation, calling the effect real rather than a statistical artifact. The distinction matters: if config changes are individually more dangerous, the fix is to make each change safer. If they are just more frequent, the fix is to reduce how often they happen.
War story
A million hypervisors, five minutes
Ian Nowland, formerly SVP of Core Engineering at Datadog and before that at AWS EC2, argues that configuration has little that distinguishes it from code in how it affects system behavior, and should get the same testing and deployment controls.
- Config pushed to S3
- Cron fetches on every hypervisor
- 1M+ updated in <5 min
- Agent bug triggered
- Near-global outage
At AWS EC2, teams built ad-hoc cron jobs that fetched config files from S3, bypassing every deployment safety control. In one near-miss, a junior engineer pushed a configuration change to over 1 million hypervisors worldwide in under 5 minutes. The change tripped a bug in an adjacent agent running on every hypervisor, causing crashes. Nowland called it very close to a global outage of new network connectivity to every EC2 instance in the world.
The workarounds
Fast propagation, not safe propagation
The AWS pattern is not unique. When deployment tooling is too idealistic to accommodate real-world needs, engineers build workarounds that become massive outage footguns. Central config stores that propagate values quickly to every node are fast, but they lack the staggered rollout, canary testing, and rollback that code pipelines provide.
Config store / S3Cron fetch on every nodeAll nodes, no gates
Nowland's takeaway is that central configuration stores used as a fast-propagation mechanism for values that change system behavior are fast, but not safe. The fix is to treat config like code: version control, staging, canary, rollback. The teams that do this are still catching up to where code deployment was years ago.
Closing the gap
Give config the code treatment
Why it matters
The observation that configuration changes rival code changes as incident causes exposes a structural gap in how systems ship changes. A decade of deployment safety tooling went into code pipelines with canaries, staged rollouts, and automated rollback, while configuration got fast propagation instead. The teams closing this gap, like Salesforce building staggered DNS rollout, are still catching up to where code deployment was years ago. Pushing config through the same version-controlled, staged, canary pipeline as code is the obvious fix, and it is still the exception rather than the norm.
The short version
- Configuration changes show up in production incidents at rates comparable to code changes, despite receiving a fraction of the safety tooling.
- Deployment pipelines built for code (canaries, stagger, rollback) were never extended to config with the same rigor.
- Config is hard to unit-test because the difference between test and production environments is the configuration itself.
- A single config push at AWS EC2 reached over 1 million hypervisors in under 5 minutes, nearly causing a global network outage.
- The fix is structural: route config through the same version-controlled, staged, canary deployment pipeline as code.