Achieve State-of-the-Art Inference Latencies with Speculative Decoding

Autoregressive decoding produces one token per forward pass, and each pass has to stream every one of the model's billions of parameters through the GPU. That makes generation memory-bandwidth bound, not compute bound: the arithmetic units sit mostly idle waiting on weights to arrive from HBM.

So the GPU has spare compute on every step, but the sequential nature of decoding gives it nothing to do with it. Speculative decoding is the trick that fills that slack.

The move is to split generation into two jobs. A cheap mechanism proposes several candidate future tokens in sequence, then the expensive target model verifies all of them in a single forward pass. That verification pass is structurally the same as prompt processing, which GPUs already run efficiently in parallel across many positions.

As long as the target accepts more than one token per verification pass on average, you spend fewer sequential target steps per output token, and wall-clock latency drops. The guarantee that makes it usable: the output distribution is provably identical to plain decoding from the target. It is lossless.

In the classic setup from Leviathan et al. and Chen et al. (2023), a small draft model autoregressively proposes K tokens, usually 4 to 8. The target then scores all K+1 positions at once and applies rejection sampling: at each position it compares the target's probability against the draft's, accepting while the target's is at least as high, and on the first rejection it discards that token and everything after it, resampling from an adjusted distribution.

If all K are accepted, the target's own forward pass also emits a free bonus token. Even the worst case, zero draft tokens accepted, still yields one token, so speculative decoding is never slower per step, it only adds proposal overhead. One hard requirement: draft and target must share the same tokenizer and vocabulary, which constrains which small model you can pair with a given target.

NVIDIA's TensorRT-LLM numbers on a DGX H200 make the gain concrete. With Llama 3.1 405B as the target on 4×H200, baseline decoding runs at 33.46 tok/s. Add a Llama 3.2 3B draft and it jumps to 120.75 tok/s, a 3.61x speedup.

Notice the shape of it. The tiny 1B draft gives a bigger speedup than the 8B draft, because a larger draft costs more latency per proposal than the extra acceptance rate buys back. For the 70B target on a single H200, the 1B draft is the clear winner at 2.86x (146.05 vs 51.14 tok/s).

The UW-Madison study "Decoding Speculative Decoding" ran over 350 experiments against LLaMa-65B and OPT-66B targets. Its first finding: throughput is bottlenecked by draft-model latency, not by token acceptance rate alone. And draft latency is driven by model depth, the number of layers, more than by parameter count, since layers must run in sequence.

The second finding is stranger: a draft model's benchmark accuracy on MMLU or Hellaswag does not correlate well with its acceptance rate. Off-the-shelf small models, tuned to squeeze accuracy per parameter, make poor drafts. Rebuilding them shallow and wide at the same parameter budget gave 60%+ higher throughput in greedy decoding and up to 111% in sampling.

A separate draft model means a second set of weights, a second KV cache, and a shared-vocabulary constraint. The alternative is to generate proposals from the target's own internals. EAGLE attaches a lightweight head, a stripped-down decoder layer plus a linear layer, that extrapolates from the target's hidden states just before the output head. EAGLE-3 fuses low, middle, and high-level features into that head.

Instead of one linear guess, EAGLE builds a dynamic draft tree: multiple candidate branches verified together with tree attention, pruning the invalid ones. The head is instance-adaptive, drafting longer on predictable text and stopping a branch once its confidence drops. Reported results land around an 80% acceptance rate and 2.5x speedups.

Medusa takes a simpler route: extra linear heads on the final hidden state, where head k predicts the token k positions ahead, forming a tree verified with a sparse attention mask. DeepSeek's Multi-Token Prediction uses several independently trained heads in the same spirit, keeping the longest matching prefix each step.

Sometimes you need no model at all. N-gram / prompt-lookup decoding builds n-grams from the prompt and generation so far, and when the current context matches something seen before, it proposes whatever followed last time. On tasks with heavy input-output overlap, summarization, doc QA, code editing, this is almost free and very effective.

vLLM reports 2.8x on CNN/DailyMail summarization at QPS=1 with prompt lookup. There is also self-speculation, where the target drafts for itself: SWIFT skips intermediate layers of the same model (1.3x, 1.6x), and early-exit methods use a partial run as the draft. The catch is early exit still runs roughly a quarter of the model, capping its ceiling far below a true 100x-smaller draft.

The algorithm is only half the work. Verification grows the target's per-step batch by (num draft tokens + 1) to score every continuation at once, so its compute cost rises faster than plain decoding. That is fine while the GPU has idle capacity, but once the system is already compute-bound at high QPS, the overhead stops overlapping with slack and starts costing you.

vLLM saw its Llama3-70B draft setup swing from 1.5x faster at QPS=1 to a 1.4x slowdown at high QPS. Getting the win in production means reusing the KV cache across draft and target calls, sequencing the two models so they do not fight over the GPU, and watching draft tensor-parallel layout, since a shrunk allocation on one GPU limits the whole TP group.