What it's for
- Latency: speculative decoding is why a chat can feel instant even though each token technically needs the whole big model.
- Cost: fewer big-model passes per answer means the same GPU serves more conversations — it stacks with the KV cache and continuous batching from the earlier chapters.
Good to know
Verification is parallel because the big model can score a whole drafted span in one forward pass — checking is not generating. This simulation shows the greedy case: each draft is the drafter's best guess and the big model's own token replaces the first miss, so the text is exactly what greedy decoding would say. With random sampling, a corrected accept/reject rule (in the references) keeps the output distribution exactly the big model's.
How it works, step by step
Why the steps go in this order
- Plain decode is serial: one expensive pass per token, and most of that pass is spent re-reading weights, not thinking.
- Most tokens are easy — a tiny model guesses them right most of the time, and guesses are cheap.
- One big-model pass over the drafted span verifies every guess at once, because scoring a given span needs no new generation.
- Keeping the longest correct prefix and falling back at the first miss preserves the big model's exact output — the speedup is real, the text is unchanged.
Before you rely on it
Four draft chips stand in for a tunable lookahead, and the acceptance rate and tokens-per-pass numbers are illustrative — real speedups depend on the drafter, the workload and the hardware. The parallel verify, the keep-the-matches rule and the exactness guarantee are exactly as shown.