Deep Dive · Issue 21

How an LLM Server Stalls After 200 OK

The server returns 200 OK, then silence. What breaks between the header and the first token.

7 October 2026 A phone showing a 200 OK response with a frozen stream

The Problem

200 OK Opens a Stream, Not a Guarantee

A phone showing a 200 OK response with a frozen stream

In a streaming LLM endpoint, the server sends 200 OK to establish the HTTP stream before prefill or generation begins. The status means the server accepted the request and opened a channel. It says nothing about whether tokens will flow.

Between the header and the first token, the request passes through gateway overhead (10-25ms), queue wait (target <50ms), CPU tokenization (8-15ms for a 1,024-token prompt), and GPU prefill (target <280ms). A stall in any of them produces silence on a connection that already returned success.

The 200 OK fires at the top. Every stage below it can still stall.

The client has no visibility into any of this. From the outside, the connection is open, the status is success, and the bytes are simply not arriving.

The Mechanism

When the KV Cache Fills, Everything Waits

GPU KV cache nearly full with requests waiting

The KV cache holds the attention key-value tensors for every active request in GPU memory. When request rates climb, the cache fills, and the system hits its saturation point: there is not enough memory to schedule new prefills or continue decoding the running batch. Incoming requests are forced to wait.

Saturation point — The load at which available GPU memory can no longer schedule new prefills or sustain ongoing decoding. Beyond it, throughput collapses and queueing latency spikes.

Under capacity

new prefills scheduled, decode continues

At saturation

no free slots, requests queue indefinitely

What the saturation point actually measures

The saturation point is defined in terms of KV cache memory, not raw GPU compute. A system can have spare FLOPs and still be saturated, because the bottleneck is the memory needed to hold attention keys and values for every token in every active request's context. Once that memory is exhausted, the scheduler cannot admit new prefills without evicting or preempting running requests, and throughput collapses.

The wait compounds through head-of-line blocking. Every queued request sits behind the ones already in flight, and the queue grows faster than the scheduler can drain it. At 70% utilization with Poisson arrivals, the P99 queue wait is roughly 3x the average, so tail latency explodes well before the system is fully saturated.

The Numbers

0.0 Tokens/s and Not a Single Error

A server rack with all status lights dark

When the deadlock hits, the symptoms are precise and silent. GPU utilization drops to zero, the server keeps accepting connections, and every request to /v1/chat/completions times out. In one case on vLLM, a multimodal model on an L4 GPU processed 1-2 requests successfully, then entered this state permanently.

Avg prompt throughput: 0.0 tokens/s,
Avg generation throughput: 0.0 tokens/s,
Running: 0 reqs, Waiting: 0 reqs,
GPU KV cache usage: 0.0%

vLLM log output during a deadlock, repeating indefinitely

Throughput drops to zero and the error log stays empty.

In a separate case, KV cache usage climbed to 98.7% and the server hung for 10 minutes without throwing a single error. Both patterns share the same signature: the API is up, the GPU is idle, and no alarm fires.

The Tradeoff

When the Model Thinks Through Its Entire Budget

Token budget consumed by reasoning tokens

A different kind of silence: the stream completes normally, but the response is empty. Reasoning models generate internal chain-of-thought tokens before producing visible output. If max_tokens is set too low, the entire budget is consumed by reasoning_content, and the API returns an empty message.content with finish_reason: "length". With gpt-oss-120b on vLLM, a max_tokens of 300 yields an empty response; set it to 1,000, and the model finishes its reasoning with room for actual output.

With max_tokens = 300, reasoning eats the entire budget. At 1,000, there is room for visible output.

The Mid-Stream Stall

Even when tokens are flowing, a stall can land mid-stream. For interactive chat, an inter-token latency P99 above 50ms causes visible stuttering; for voice agents, the budget is 30ms. A single 1-second gap mid-stream is enough to make the user think the connection died.

Wrap-up

200 OK Is Not a Health Check

Why it matters

The gap between HTTP success and token delivery is a class of failure that standard monitoring misses. A health check that probes the endpoint and checks for 200 OK reports green while the server is deadlocked, the KV cache is full, or the model spent its budget on internal reasoning. The systems most exposed are services that treat the LLM endpoint like any other HTTP API, where a green status page coexists with a completely stalled inference pipeline.

The short version

`200 OK` opens the HTTP stream before prefill or generation begins; it says nothing about whether tokens will flow.
KV cache exhaustion pushes the system past its **saturation point**, where new requests queue indefinitely behind head-of-line blocking.
vLLM can deadlock with **0.0 tokens/s** and zero GPU utilization, accepting connections while processing nothing and throwing no errors.
A reasoning model with `max_tokens` set too low returns an empty response with `finish_reason: "length"` because the entire budget was consumed by internal thinking.
Mid-stream ITL spikes above **50ms** cause visible stuttering; a 1-second gap makes the user think the connection died.