System Design · Issue 15

The leak only production traffic could expose, and the dependency chain that blocked rollback

Mistral's vLLM setup leaked 400 MB/min in a way no profiler could see, and the fix was trapped two layers below the code they owned.

5 September 2026 vLLM serving cluster with memory climbing on decode nodes

The constraint

A leak that hid from every test

vLLM serving cluster with memory climbing on decode nodes

Mistral AI was pre-production testing disaggregated serving with their frontier model Mistral Medium 3.1 on vLLM, with graph compilation enabled. Under production-like traffic, system memory climbed at 400 MB per minute, linear and steady, until the process hit OOM a few hours later.

The leak appeared only on the decode side of a Prefill/Decode split using NIXL for KVCache transfer over InfiniBand. Smaller models, fewer optimizations, and standard reproduction attempts all failed to trigger it. The vLLM team confirmed the issue via GitHub, so it was not environment-specific.

Prefill/Decode disaggregation: the leak lived on the decode side, where KVCache arrives via NIXL over InfiniBand

The leak required the full stack: a large model, graph compilation, and the NIXL transfer path over InfiniBand. Nothing less would trigger it.

The blind spot

Heaps do lie

Profiler examining a stable heap while anonymous mappings grow below

The team reached for the standard toolbox. Memray showed no leak. Guppy 3 showed no leak. GDB crashed the process. Valgrind was too slow for vLLM's heavyweight setup. Heaptrack, a malloc/free tracer, reported a stable heap. The process RSS kept climbing anyway.

Heaptrack hooks glibc's malloc and free, tracking the traditional heap but missing memory allocated through direct mmap calls or mremap. RSS encompasses the heap, the stack, and anonymous memory mappings, and a leak in the last category is invisible to anything that only watches the heap.

Heap (malloc/free)

stable, tracked by Heaptrack

Anonymous mappings (mmap)

growing, invisible to profilers

The blog title says it plainly: heaps do lie. The Python heap was clean. The leak was somewhere else entirely, in a memory region no Python-level tool could see.

The breakthrough

Anonymous memory, changing addresses

pmap output showing growing anonymous memory regions with changing addresses

The team ran pmap -X $pid every second and sorted the output by RSS size. Most memory regions stayed static, but a few anonymous mappings kept growing. Their start addresses were changing between snapshots, the signature of mremap, a syscall that resizes or relocates memory regions without freeing them.

The leak traced down a four-layer dependency chain. vLLM called NIXL for KVCache transfer. NIXL called UCX (Unified Communication X) for the actual InfiniBand data path. UCX allocated anonymous memory through mmap and mremap at the kernel level. vLLM's Python heap was clean the entire time.

The dependency chain: the leak lived in UCX's memory management, two layers below vLLM

Why mremap changes addresses

mremap is a Linux syscall that resizes or relocates a memory region in the process address space. When a region grows beyond its current virtual address range, the kernel may relocate it to a new address, which is why the team saw start addresses shifting between pmap snapshots. The old mapping is freed and a new one created, but if the old region's contents were never properly released, the relocation pattern masks the leak from any tool that only tracks malloc and free.

The rollback trap

No layer can be fixed alone

Stacked dependency layers with the bug trapped between layers and rollback blocked at each boundary

The bug was in UCX, a system-level communication library two layers below vLLM's Python code. The blog frames this as "the hidden risks of dependencies layers in today's software." Rolling back vLLM alone would not help because the faulty allocation path runs through NIXL into UCX's memory management, below anything Mistral could patch.

Patching UCX was the obvious move, but NIXL depends on a specific UCX API surface. Changing UCX risks breaking the NIXL integration that vLLM's disaggregated serving requires, the structural trap that circular dependencies create in deployment graphs.

The bug was two layers below the code anyone could read, in a library nobody on the team owned.

Both rollback paths hit a wall: the bug is below vLLM, and the fix risks breaking NIXL above it

The dependency chain that made the bug hard to find also made the fix hard to ship. Every path forward touched a layer the team did not own, and every change risked the layer above or below it.

Takeaway

Where production-only leaks live

Why it matters

The vLLM leak shows how modern inference stacks accumulate hidden risk. A serving framework pulls in a transfer library that pulls in a communication library that talks to the kernel, and the bug surface extends far below the code any team writes or tests. Production traffic is the only thing that exercises the full stack at the scale and configuration where latent allocation bugs surface, and the same dependency chain that makes the bug hard to find also makes the fix hard to ship without breaking something else.

The short version

Mistral's vLLM setup leaked **400 MB/min** under production-like traffic, hitting OOM in a few hours.
Every Python-level profiler showed a clean heap because the leak was in **anonymous memory mappings** allocated via `mmap` and `mremap`, invisible to malloc hooks.
`pmap -X` revealed growing anonymous regions with **changing addresses**, the signature of `mremap` relocating without freeing.
The leak lived in **UCX**, two dependency layers below vLLM's Python code.
The dependency chain blocked both rollback and patching: the bug is below vLLM, and fixing UCX risks breaking the NIXL API above it.