System Design · Issue 21

Inside ConServe's 84× Latency Cut for Co-Served LLMs

How UCLA and Berkeley built a scheduler that co-serves batch and real-time LLM inference on the same GPUs without collapsing online latency.

7 October 2026 GPU servers mostly idle while a few handle a traffic spike

The Problem

70% of GPU Cycles Bought and Wasted

GPU servers mostly idle while a few handle a traffic spike

LLM inference clusters are provisioned for peak. At moderate load around 2000 tokens/s, more than 70% of GPU resources sit idle on an H100 running Llama-3.1 8B.

The wasted cycles could absorb offline batch work: document summarization, data analytics, the jobs that batch APIs already run with 24-hour turnaround. Same model weights, same inference engine, no process switching.

Why nobody co-serves today

The failure chain of naive co-serving with existing engines

Naive co-serving makes online latency worse. A single offline batch iteration can block online requests for up to 970ms on Llama-3.1 8B, and Sarathi-Preemptive pushes P99 time-to-first-token up 3.59× versus online-only.

The Design

A Cost Model at Every Iteration

Scheduler checking preemption flag at layer boundaries

ConServe replaces static chunk budgets with a learned cost model. A one-time profiler fits a polynomial that predicts iteration latency from P (active compute tokens) and C (cached context tokens). At runtime, the scheduler queries this model every iteration to find the maximum offline tokens it can admit without breaching online SLO slack.

\text{Latency}(P, C) = k_1 P + k_2 P(P+C) + k_3 P + k_4 (P+C) + k_5

Polynomial cost model: P = active compute tokens, C = cached context tokens. Terms cover linear compute, quadratic attention, communication, memory access, and constant overhead.

Preempting between layers

Existing engines run a full forward pass as one uninterruptible CUDA Graph. ConServe inserts safepoints between transformer layers, each executing in a few milliseconds, so workers can check a preemption flag at each boundary, drop offline requests, and continue with online ones only. Preemption latency drops from hundreds of milliseconds to single-digit milliseconds.

Tensor-parallel synchronization

In multi-GPU deployments, workers must agree on when to preempt. ConServe uses ncclBroadcast to synchronize all workers at each safepoint, preventing deadlocks and maintaining CUDA Graph compatibility. The overhead when no preemption occurs is <1.1%.

The Missing Piece

Checkpointing KV Cache One Token at a Time

Incremental KV cache checkpoint of one token to host memory

KV cache is append-only. Once generated for a token, it never changes. ConServe exploits this by checkpointing only the single new KV cache entry produced each decode iteration to host memory, asynchronously. Transfers complete in ~7ms and fully overlap with computation, so the I/O cost is effectively zero. On resume after preemption, evicted pages are prefetched in the background.

Incremental KV cache checkpointing. Per-iteration I/O is bounded by one token, independent of sequence length.

What each component buys

The ablation on Llama-3.1 8B tells the story. Starting from a preemptive vLLM baseline at 375ms P99 time-between-tokens, adding the SLO-aware scheduler alone drops it to 73ms. Layer-wise preemption cuts P99 time-to-first-token further to 523ms with <1.1% overhead when idle. Incremental KV cache management adds a 1.29× offline throughput boost by eliminating recomputation on resume.

The Tradeoff

Where the Numbers Land

Tradeoff between offline throughput and online latency SLOs

The cost model makes the tradeoff explicit: tighter SLO slack means fewer offline tokens admitted per batch. With a modest 1.5× SLO scale factor, ConServe hits 93% of optimal offline throughput while keeping >99% SLO attainment.

The original v1 paper reported an 84× latency reduction against existing co-serving systems on Llama-2-7B. The revised ICML 2026 version, re-evaluated against SOTA preemptive systems on Llama-3.1 and H100, reports 2.9× tail latency reduction and 2.2× offline throughput.

The design has real limits: per-model profiling, model-specific safepoint instrumentation, and host memory as a backing store for checkpointed KV cache. It assumes online and offline share the same model weights, so co-locating with training is out of scope.

Wrap-Up

Co-Serving Without the Collapse

Why it matters

ConServe shows that the barrier to GPU co-serving was never the hardware. It was that serving engines treated a full forward pass as atomic and a request's KV cache as an inseparable blob. By cutting preemption to layer granularity and checkpointing cache one token at a time, the system makes the idle 70% of inference GPUs usable without breaking online SLOs. For teams running LLM inference at scale, the design offers a concrete path toward halving their GPU bill.

The short version

More than **70%** of GPU resources sit idle at moderate LLM inference load because clusters are provisioned for traffic spikes.
ConServe co-schedules offline batch inference on the same GPUs, using a learned cost model to admit offline tokens only when online SLO slack allows.
Layer-wise preemption cuts preemption latency from hundreds of milliseconds to **single-digit milliseconds** by checking a flag between transformer layers.
Incremental KV cache checkpointing copies one token's cache to host per iteration in **~7ms**, fully overlapped with computation.
Revised numbers are **2.9×** tail latency reduction and **2.2×** offline throughput against SOTA preemptive systems; the original 84× was measured against weaker baselines on older hardware.