Paper · Issue 13
The Benchmark Lie: How Rearranging Environment Variables Can Flip Which Compiler Wins
Same code, same compiler, same input, different winner. A 2009 paper showed an unused environment variable can swing a benchmark by almost 300%.
The setup
The benchmark that changes its mind
Compile a program two ways, time both, keep the faster flag. That loop sits under almost every performance claim in systems research. In 2009 four researchers showed you can flip the winner of that comparison without touching the code, the compiler, or the input, just by padding an unused environment variable to a different size.
env ≈ 0 bytes
-O3 looks faster
env ≈ 4 KB
-O2 looks faster
Measurement bias — A systematic, direction-flipping tilt baked into how an experiment is set up, different from random run-to-run noise. It can overstate an effect, or invert the conclusion entirely.
The unnerving part is that nothing in the experiment looks wrong. Every number is measured correctly, the loop runs to completion, the timer is honest. The bias lives in a knob nobody thought to hold fixed.
The mechanism
Where your stack starts depends on your shell
The trick is not magic, it is geography. The UNIX environment is loaded into memory before the call stack, so its total size sets the address where the stack begins. Add a few hundred bytes of unused variable and the whole stack slides, which changes how stack-allocated locals line up inside caches and other hardware structures.
high ┌────────────────┐
│ argv + env │ ← size varies
├────────────────┤
│ call stack │ ← start address shifts
│ i j k │ ← alignment moves
low └────────────────┘The effect is so subtle it can ride in on almost anything that ends up in the environment. Gabriel Scherer's benchmarking guide points newcomers to this paper as a scary story: two runs differing only in their UNIX username, and therefore in a few characters of string length, produced a 10% performance gap.
The demonstration
One unused variable, 600K to 1.6M cycles
To isolate the effect they wrote the smallest program that could show it: a tight loop over three globals, with optimizations disabled so the compiler could not delete the loop as dead code.
static int i = 0, j = 0, k = 0;
int main() {
int g = 0, inc = 1;
for (; g < 65536; g++) { i += inc; j += inc; k += inc; }
return 0;
}On a Core 2 workstation, sweeping the added environment size from roughly 0 to 4000 bytes, runtime swung from about 600,000 to about 1,600,000 cycles. Each point is the mean of 5 runs with 95% confidence whiskers, so this is not jitter. The measured bias ran about 33% typically and once nearly 300%, from a variable the program never touches.
The same class of bug is alive and well in modern toolchains. A .NET writeup shows two functions doing identical work landing about 2x apart purely because of where the JIT places their code in memory.
The real experiment
Rearranging the setup flips the winner
The micro-loop is a toy. The real study asked a question people actually care about: is gcc's -O3 a speedup over -O2? They defined it cleanly as runtime(O2) / runtime(O3), ran the C programs of SPEC CPU2006, and then varied two incidental knobs, environment size and the linker's order of .o files.
Setup A
-O3 wins
Setup B
-O2 wins
Depending only on those layout choices, -O3 came out a win, a loss, or a wash. They replicated it on a Pentium 4, a Core 2, and an m5 simulator, and with a second compiler, Intel's ICC, so it is not a quirk of one chip or one toolchain. Worse, the best link order on one processor was often not the best on another, so there is no universal safe setup to standardize on.
How common is it
133 papers, zero safeguards
If this only bit toy loops it would be a curiosity. So they audited 133 recent papers from ASPLOS, PACT, PLDI, and CGO, the top venues for exactly this kind of measurement. Of the ones with experimental results, none adequately accounted for measurement bias.
papers surveyed ████████████████████ 133 handled bias ▏ 0
Averaging over a benchmark suite only cancels bias if the suite is diverse enough, and they showed SPEC CPU2006's C programs are not, echoing the DaCapo group's earlier finding that SPEC JVM98 was not representative of real Java workloads. Scherer's methodology guide now lists this paper as required reading before you trust any benchmark number.
The fix
Randomize the layout, or there may be nothing to measure
Their fix borrows from how natural sciences fight bias. Instead of trusting one incidental setup, run the experiment across many deliberately different setups and summarize the distribution of outcomes, then use causal analysis to check whether an effect survives once layout is accounted for.
layout 1 [fn_a][fn_b]… → 0.61M layout 2 [fn_b]…[fn_a] → 1.40M layout 3 random → 0.92M ⋮ ⋮ many random layouts → ▁▂▄█▄▂▁ (normal)
Why a normal distribution is the point
Re-randomizing placement many times makes each run an independent draw, so by the central limit theorem the runtimes cluster into an approximately normal distribution. That is what finally lets you apply ordinary statistical tests like ANOVA and get an answer that means something.
STABILIZER took this to its conclusion, continuously re-randomizing the placement of functions, stack frames, heap objects, and globals during execution. When they applied it plus ANOVA across SPEC CPU2000 and CPU2006, LLVM's -O3 over -O2 came out indistinguishable from noise. Remove the layout bias, and a flag everyone trusts may have no measurable effect at all.
The takeaway
Trust the distribution, not the run
Why it matters
Almost every performance claim in systems work rests on the same compare-two-configs loop, and this paper showed that loop can be silently reordered by things nobody thinks to hold fixed, like the byte-size of an unused environment variable. That reframes a single benchmark run from evidence into an anecdote. If you benchmark to pick a compiler flag, a library, or a data structure, one clean-looking number under one setup can point you the wrong way, and the honest answer often requires many randomized layouts before an effect is real.
The short version
- An unused environment variable shifted a program's runtime from ~600K to ~1.6M cycles, biasing results by about **33% typically and nearly 300% once**.
- The cause is memory layout: the environment loads before the stack, so its size changes how variables align in caches.
- Varying only env size or linker order flipped whether **gcc -O3 beat -O2** across SPEC CPU2006, on multiple chips and compilers.
- Of **133 top-venue papers** surveyed, none adequately handled this bias.
- STABILIZER randomized layout and found **-O3 vs -O2 indistinguishable from noise**, so remove the bias and the effect may vanish.