Deep Dive · Issue 6

Speculation Is All You Need: How Speculative Decoding Wins the Latency Budget

Decode is sequential and memory-bound, so the GPU sits half-idle. Speculative decoding fills that slack with guessed tokens, for multiplicative speedups.

5 August 2026 Latency broken into network, host overhead, prefill, and a dominant decode stage

The bottleneck

Decode is where the time goes

Latency broken into network, host overhead, prefill, and a dominant decode stage

Serving latency splits into four additive pieces: client-to-server network, host and orchestration overhead, prefill, and decode. The first three you can shave by percentages. Decode is the one that dominates, and it behaves differently from the rest.

Decode is inherently sequential: one full weight load and forward pass per output token. That makes it memory-bandwidth-bound, not compute-bound, so at small batch sizes the GPU's arithmetic units sit mostly idle while weights stream in from memory.

  prefill        decode (sequential)
  ┌─────┐   t1   t2   t3   t4   t5 ...
  │█████│ → □ → □ → □ → □ → □ → ...
  └─────┘  one weight-load + forward pass per token
           arithmetic units: mostly idle

Each decode step reloads the full model to emit a single token.

That idle arithmetic bandwidth is the opening. If there is slack to run several tokens through a forward pass for roughly the cost of one, you can trade unused compute for fewer sequential steps, which is exactly what speculative decoding does.

The core trick

Guess a few tokens, then check them all at once

A small draft model proposes tokens, a large target model verifies them in parallel

A small, cheap draft model proposes γ tokens autoregressively. The large target model then verifies all γ in a single forward pass, which is shaped like prefill and runs in parallel rather than one token at a time.

Draft proposes γ tokens (cheap, autoregressive)
Target verifies all γ (one parallel pass)
Accept prefix, resample rest (rejection sampling)

The clever part is that it stays lossless. Each drafted token is accepted with probability min(1, p_target/q_draft), and on the first rejection you resample from the corrected distribution. The output distribution is provably identical to plain decoding from the target model, so speed costs you no quality.

Theory vs. wall clock

Why the textbook speedup rarely shows up

Predicted speedup far ahead of measured speedup

The classic model says speedup depends on three numbers: acceptance rate α, drafted tokens per step γ, and the draft/target cost ratio c.

\text{speedup} = \frac{1 - \alpha^{\gamma+1}}{(1-\alpha)(\gamma c + 1)}

Theoretical speedup from the original speculative-decoding analysis.

Reality bites. On an A100, a llama-160m draft for a llama-7B target with γ=5 measured α=0.448, predicting a 1.69× speedup. The actual result was 0.66×, slower than baseline (50 vs 76 tokens/s), and even forcing α to ~0.99 gave only 1.18× against a theoretical 5.27×. Sampling, kernel launches, and host synchronization eat the paper gains.

There is a subtler trap too: a higher-acceptance draft is not automatically a faster one, because a bigger draft is also slower per step. Draft latency and acceptance have to be tuned together, which is why a custom 1.3B draft for LLaMA-65B beat existing drafts by 30% on throughput while training on just 0.42B tokens.

Better drafts

From separate models to feature-level trees

Extra heads on a target model branching into a tree of candidate tokens

Medusa drops the separate draft model entirely: it bolts extra decoding heads onto the frozen target's last hidden state, each predicting a further-future token, then verifies a Cartesian tree of candidates with a sparse attention mask. With next-next-token top-5 accuracy above 80%, the tree of guesses reaches 2×, 3× speedups.

EAGLE-2 drafts at the feature level, reusing the target's own representations before the LM head, and reshapes its draft tree per context using the draft model's well-calibrated confidence scores. That pushes it to 3.05×, 4.26×, roughly 2× faster than Medusa, and still fully lossless.

         hidden feature
              │
         draft head
         /    |    \
       t_a   t_b   t_c      ← top-k per step
      / \    |    / \
    ...  ... ...  ...       ← dynamic tree, verified in one pass

A context-aware draft tree lets one verification pass check many candidate continuations.

The newest step, DFlash, reads the target's KV projections directly and generates draft tokens in parallel rather than autoregressively, fitting hardware with high arithmetic intensity. A DFlash speculator for Qwen 3.5 397B-A17B improves acceptance and performance over generic multi-token-prediction heads by more than 50%.

The biggest lever

Match the draft to your actual task

Generic draft versus a task-tuned draft feeding a production service

Off-the-shelf speculators disappoint in production because their output distribution does not match your task. Real EAGLE-3 drafts on vLLM and SGLang delivered only ~2× with acceptance rates stuck in the 0.6-0.8 range, far below the near-ideal numbers used in simulations.

The fix is to train the draft on production-like data. Modal's Decagon Voice case study started at ~290ms p50; the full low-latency playbook plus a custom DFlash speculator cut ~100ms, with the speculator alone removing about 40% of server-side decode latency.

  public / aligned prompts
           │
     ┌─────▼─────┐
     │  TARGET   │  →  in-grammar completions
     └───────────┘        │
                          ▼
              train custom draft model
              (hold out data → track
               acceptance on real traffic)

"Mid-training": synthesize task-matched data by decoding through the target itself.

You do not even need sensitive data. Pass public or loosely-aligned prompts through the target model to generate completions in the same output grammar, which is ideal for strict structured or tool-calling tasks, then hold out data to guard against overfitting and confirm acceptance length generalizes back to live traffic.

Tuning the knobs

How big should the draft be?

A tradeoff scale between a fast small draft and an accurate larger draft across GPUs

The Dovetail performance model exposes the real knobs: γ raises verification cost linearly, a heavier draft raises its own latency, and acceptance α buys you longer accepted runs. Net cost is the ratio of drafting-plus-verifying time to the expected number of accepted tokens.

\frac{T_{\text{SD}}}{T_{\text{target}}} = \frac{1}{\Omega(\gamma,\alpha)}\left(\gamma\,\frac{T_{\text{draft}}}{T_{\text{target}}} + \frac{T_{\text{verify}}(\gamma)}{T_{\text{target}}}\right)

Ω(γ,α) is the expected count of accepted tokens per step.

Hardware decides the sweet spot. EAGLE-2 with γ=60 hit Ω=5.59 and 4.01× on an RTX 4090, but the same config on a weaker RTX 2080 managed only 1.90× as verification cost dominated; dropping γ to 16 recovered it to 2.32×.

Which is why smaller can win. Early systems used drafts with ~50× fewer parameters than the target, and later work pushed to 528× fewer for models like LLaMA-65B. A bigger, more accurate draft can still reduce net throughput once its per-step latency outweighs the extra tokens it earns you.

The payoff

The multiplicative lever

Why it matters

Decode dominates LLM serving latency and only responds to speedups that cut the number of sequential steps, which is what makes speculative decoding stand apart from the percentage-level wins available in kernels and infrastructure. The gains are real but conditional: they show up only when the draft is tuned to the target and task, γ is sized to the hardware, and the serving stack keeps host overhead low. For teams chasing tail latency, training a custom speculator on task-matched data is now the highest-leverage move available.

The short version

Decode is sequential and memory-bandwidth-bound, so the GPU has spare compute that speculative decoding fills with guessed tokens verified in one pass.
Rejection sampling keeps the output **lossless**, identical to plain decoding from the target model.
The textbook speedup formula overstates reality: real overhead turned a predicted 1.69× into a measured **0.66×** on one A100 test.
Feature-level, tree-structured drafts (EAGLE-2, DFlash) reach 3×–4×+, but a task-tuned draft matters more than a bigger one.
Training a custom speculator on production-like data removed ~40% of server-side decode latency in Modal's Decagon case study.