How Burst Traffic Kills LLM Serving

LLM inference runs in two phases. Prefill computes the attention for an entire prompt in one shot and is compute-bound, eating GPU FLOPs. Decode generates tokens one at a time and is memory-bound, limited by how fast the KV cache can be read. Under steady load, a serving engine interleaves these phases comfortably. A burst of new long prompts breaks that balance.

In a prefill-prioritizing scheduler like vLLM, a burst forces the system to pause every ongoing decode to compute the incoming prefills. A 32K-token prefill can hog 80-95% of the GPU for 2-8 seconds, starving ~200 concurrent decode users who may receive only 1 token in 5-10 seconds. The GPU is busy, the users are stuck, and adding more GPUs does not help because the problem is scheduling, not capacity.

Sarathi-Serve fixes this with chunked prefills: instead of computing a 4096-token prompt in one shot, it splits the prompt into 8 chunks of 512 tokens and interleaves them with running decodes in the same batch. New requests join without pausing anyone. This stall-free scheduling achieves up to 2.6× higher throughput for Mistral-7B on A100s and up to 5.6× for Falcon-180B with pipeline parallelism.

Most serving engines schedule requests first-come-first-served with run-to-completion: once a request starts decoding, it runs until it emits its last token. Real-world LLM workloads from traces like ShareGPT and Alpaca have highly skewed, long-tail output length distributions. A few requests generate thousands of tokens while most finish in under a hundred.

Under burst traffic, a long job that arrived first blocks every short job behind it. Up to 90% of total latency can be pure queuing delay, not compute. The GPU finishes the work fast enough; the requests just sit in line.

FastServe replaces FCFS with a skip-join Multi-Level Feedback Queue that preempts at the token level, not the request level. Short jobs get boosted to a high-priority queue and bypass long ones mid-decode. The long job is paused, the short job runs, and the long job resumes when capacity frees up. Under the same average latency targets, this improves throughput by up to 31.4× compared to vLLM.

When a burst of concurrent requests arrives, each needs KV cache space. Naive allocation pre-reserves a contiguous VRAM block sized for the maximum sequence length of every incoming request, even if the actual output is short. Under bursty multi-user loads, most of that reserved space is never touched.

The waste is severe. Naive allocation burns 60-80% of the KV cache budget on reserved-but-unused memory. On a 40GB A100, this drops effective capacity from ~50 concurrent sequences down to ~20. The GPU has room for 50; fragmentation leaves it serving 20.

PagedAttention, introduced in vLLM, divides the KV cache into fixed-size physical blocks of 16 tokens allocated on demand. A request that generates 50 tokens uses 4 blocks, not a 2048-token slab. Waste drops to under 4%, and a 40GB A100 recovers its full ~50-sequence capacity.

Multi-turn agentic applications accumulate long contexts, up to 1M tokens over hundreds of turns. Because most context is reused across turns, KV cache hit rates are typically ≥95%. A burst of agentic requests means loading massive cached KV states from external storage into the serving cluster.

Standard disaggregated architectures route all this traffic through prefill engines. An H200 prefill instance can produce around 60 GB/s of KV traffic, saturating prefill-side storage NICs while decode engine NICs sit idle. The bottleneck is the network path the KV cache takes, not the compute.

DualPath introduces a dual-path loading mechanism: KV cache loads directly into decoding engines and transfers to prefill engines via RDMA over the compute network, bypassing the saturated storage NICs. This improves offline inference throughput by up to 1.87× and online serving throughput by an average of 1.96× without violating SLOs.