Paper · Issue 20

How ConServe Preempts LLM Tasks at Layer Granularity

ConServe interrupts LLM batches between transformer layers, cutting preemption latency from hundreds of milliseconds to single digits.

23 September 2026 GPU contention between online and offline LLM work

The co-serving problem

GPUs Sit Idle While Online Requests Wait

GPU contention between online and offline LLM work

Online LLM serving demands strict millisecond SLOs, but load is bursty. ChatGPT traffic can triple within seconds. Providers overprovision for peak, leaving >70% of GPU idle at average load, measured at ~2000 tokens/s average vs ~5000-6000 peak on H100 with Llama-3.1-8B.

The obvious fix is to co-serve latency-tolerant offline batch work on the same GPUs. Current engines run each batch iteration as an atomic CUDA Graph, so preemption only happens after the full iteration completes. Profiling shows a large offline batch can stall online requests by up to 970 ms.

Each offline iteration runs to completion before the GPU can serve an online request. Under saturated GPUs, the stall can reach 970 ms.

Layer-wise preemption

Breaking the Iteration at Layer Boundaries

Safepoints inserted between transformer layers

ConServe instruments transformer layers with safe preemption points between layers. Workers check a preemption flag at each inter-layer boundary. When the scheduler's monitor predicts an SLO violation, it signals preemption; workers terminate offline computation, clean up partial states, and continue with online-only requests.

Safepoint — A checkpoint between transformer layers where a worker checks whether it should preempt. ConServe inserts one after every layer, making each layer boundary a potential exit from the current iteration.

The choice of layer boundaries is deliberate. Each layer executes in only a few milliseconds, layer sizes are small and roughly uniform across LLMs, and per-layer instrumentation is lightweight. The overhead when no preemption occurs is <1.1%.

A preemption flag checked between every layer. When set, the worker cleans up partial state and yields to online requests in single-digit milliseconds.

How tensor-parallel workers stay synchronized

In tensor-parallel deployments, multiple GPUs cooperate on the same batch, each computing a slice of every layer. A preemption signal must reach all workers or some GPUs race ahead while others stop, causing deadlocks. ConServe uses ncclBroadcast to synchronize workers at each safepoint, ensuring all GPUs check the preemption flag together. This keeps the approach compatible with CUDA Graphs, which require deterministic control flow.

The full system

What the Three Mechanisms Buy

ConServe's three mechanisms feeding into a shared GPU

Layer-wise preemption is one of three mechanisms. A learned polynomial cost model replaces static chunk budgets, letting the scheduler size offline token admission to stay within online SLO slack.

\text{Latency}(P, C) = k_1 P + k_2 P(P + C) + k_3 P + k_4 (P + C) + k_5

Learned cost model: P = active compute tokens, C = cumulative context tokens in KV cache. Coefficients fit from offline profiling.

KV cache is append-only, so ConServe manages it at the token level. After each iteration, only the new token's KV entry is asynchronously checkpointed to host memory. Evicted pages prefetch in ~7 ms on resume.

On H100 with Llama-3.1 8B/70B and Qwen-2.5 14B: 2.2× higher offline throughput vs SOTA, P99 tail latency reduced 2.9×, and SLO attainment within 25% (TTFT) and 19% (TBT) of the online-only baseline.

Why layers, not kernels

The Sweet Spot Between Coarse and Fine

Granularity spectrum from coarse iteration to fine kernel, with layer boundaries highlighted

Full iterations are atomic because the CPU scheduler hands a batch to a dedicated GPU executor running a single CUDA Graph. Breaking this at the kernel level would be too expensive: instrumenting every CUDA kernel adds overhead and complexity. Layer boundaries are the natural middle ground.

Iteration-level

atomic, hundreds of ms to preempt

Layer-level

interruptible, single-digit ms per layer

ConServe is implemented as an extension to vLLM, building on continuous batching and PagedAttention. With a modest SLO slack of 1.5×, it achieves 93% of optimal offline throughput with >99% SLO attainment. Under real-world dynamic workloads, it sustains 78% of optimal offline throughput while meeting all online SLOs.

Wrap-up

Preemption at the Right Grain

Why it matters

Layer boundaries turn out to be the natural preemption point for transformer inference, small enough to respond in milliseconds, large enough to instrument cheaply. ConServe shows that the barrier to GPU harvesting was the absence of a preemption primitive that could interrupt work without wrecking throughput. Implemented as a vLLM extension, it means teams already on vLLM can in principle adopt layer-wise preemption without switching engines.

The short version

ConServe inserts **preemption safepoints between transformer layers**, cutting preemption latency from hundreds of ms to single-digit ms per layer.
Overhead when no preemption occurs is **<1.1%**, so the instrumentation is nearly free.
A learned cost model and incremental KV cache management complete the system, enabling **2.2× higher offline throughput** while meeting online SLOs.
Implemented as a **vLLM extension** using continuous batching and PagedAttention.