In this article
Before you start
This page sits in the LLM inference optimization guide and assumes the basics of the transformer architecture.
Why interviewers ask this
Generation speed decides whether a chat or voice product feels usable, so speculative decoding is a standard topic in inference and system design rounds, and it is reported by candidates in questions about cutting latency and about choosing between it and quantization. Interviewers use it to see whether you understand why generation is slow in the first place.
A weak answer says "a small model guesses and a big model fixes it." A strong answer says why the check is cheap (one pass scores many tokens), why the output is still exactly the large model's distribution, and when the method stops paying off. A Staff-level answer adds the cost model: the gain depends on how often the large model agrees and on how expensive the draft is, and it shrinks when serving is already compute-bound.
Speculative decoding from first principles
- Speculative decoding
- Speculative decoding is an algorithm that samples from a large autoregressive model faster by letting a cheaper model propose several tokens and having the large model verify them in one forward pass, using an accept-or-reject rule that leaves the output distribution exactly unchanged.
- Draft model
- A draft model is the small, fast model in speculative decoding that proposes the next few tokens. It is cheap enough to run several times per cycle and only needs to agree with the large model often, not to be as accurate as it.
In plain words
Think of a senior editor and a junior writer. Normally the editor writes every word of an article one at a time. With speculation, the junior writes the next few words quickly. The editor reads them in one glance. The editor keeps the words they would have written anyway. They fix the first one they would not. The analogy breaks in one place: the editor's check is not a loose review. It is a probabilistic rule. It makes the final text come out with exactly the same odds as if the editor had written it alone.
In one precise sentence, speculative decoding trades some cheap draft work for fewer expensive target-model passes while sampling from the target model's own distribution.
Standard generation is called autoregressive decoding: the model produces one token, appends it to the input, and runs again for the next token. Decoding K tokens takes K serial runs of the model, as Leviathan et al. (2023) put it. The target model is the large model whose distribution we must reproduce.
Step 1: The draft model proposes a few tokens
The draft model reads the text so far and generates a short run of tokens, say four (illustrative). Call that count γ. It runs the usual one-token-at-a-time loop, but it is small, so each step is cheap.
Input: the text so far. Output: γ proposed tokens, and the draft model's probability for each. Keep those probabilities: Step 3 needs them. Leviathan et al. call this the approximation model, and Chen et al. (2023) call it the draft model. You now know where the guesses come from; next is why checking them is cheap.
Step 2: Why is speculative decoding faster? One pass scores every draft
The target model receives the text plus all γ drafted tokens and runs once. A transformer computes an output at every position of its input in the same pass, so one run gives the target model's probability distribution after the original text and after each drafted prefix, γ plus one distributions in all.
This is cheap because generation is limited by memory, not arithmetic. Chen et al. (2023) observe that the latency of scoring short continuations in parallel is comparable to that of sampling a single token from the large model. The weights are read from memory once either way; see the key-value (KV) cache for the other large memory cost.
Input: text plus drafts. Output: γ plus one target distributions.
Step 3: Accept or reject each token, then resample
Rejection sampling is a way to draw samples from one distribution by proposing from another and keeping each proposal with a probability that corrects the difference. Here, draft token x has draft probability q(x) and target probability p(x). Accept it with probability . If the target likes the token at least as much as the draft did, it is always kept.
At the first rejection, stop. Throw away the rest of the drafts and sample the replacement token from the residual distribution: the positive part of p minus q, rescaled to sum to one, written . This is the term candidates forget, and it is what makes the result exact: the lost probability mass must be refilled where the target wanted more than the draft gave.
A toy case with three tokens shows it works, with a simulation of 200,000 draws.1 If all γ drafts are accepted, the target's last distribution yields one bonus token for free.
Step 4: Keep the accepted tokens and repeat
Each cycle therefore emits the accepted tokens plus one more token, either the resampled replacement or the bonus. Even if the first draft is rejected, the cycle still yields one token, the same as a normal target pass, so the worst case is one token per cycle plus the wasted draft work.
The accepted prefix becomes the new input, and the loop restarts at Step 1. Because every emitted token is a valid sample from the target's distribution, the sequence is too. Leviathan et al. (2023) report a 2 to 3 times acceleration on the T5 extra-extra-large (XXL) model with identical outputs, and Chen et al. (2023) report 2 to 2.5 times on Chinchilla, a 70 billion parameter model, in a distributed setup. The numbers are theirs and depend on the model and hardware.
Step-by-step description
- The draft model proposes a few tokens.
- The target model scores them in one pass.
- Accept each token, resample at the first rejection.
- Keep the tokens and start the next cycle.
- A small draft model proposes several tokens.
- The target model scores all drafted tokens in one pass.
- Each token is accepted or rejected, and a rejection is resampled from the residual.
- Each cycle emits the accepted tokens plus one more token, then repeats.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
The toy case has three tokens, with draft probabilities (0.6, 0.3, 0.1) and target probabilities (0.4, 0.4, 0.2), all illustrative.1 The acceptance probabilities are 0.6667 for the first token and 1 for the others. The chance a drafted token is accepted overall is 0.8, the sum of the smaller of each pair. The residual is (0, 0.5, 0.5). The final probabilities come out at exactly (0.4, 0.4, 0.2), the target's, and a simulation of 200,000 draws gave 0.4016, 0.3983 and 0.2001.1
Now the speed. If each drafted token is accepted independently with probability α, the expected number of tokens per cycle with γ drafts is the geometric sum . With α of 0.8 and γ of 4 that is 3.3616.1 The draft costs time too. If one draft step costs a fraction c of a target pass, a cycle costs target passes, so the speedup is the tokens per cycle divided by that. At c of 0.05, the speedup is 2.8013.1
| Quantity | Value (computed, inputs illustrative) |
|---|---|
| Tokens per cycle, α 0.5, γ 4 | 1.9375 |
| Tokens per cycle, α 0.8, γ 4 | 3.3616 |
| Tokens per cycle, α 0.9, γ 4 | 4.0951 |
| Speedup, α 0.3, c 0.05, γ 4 | 1.1876 |
| Speedup, α 0.8, c 0.05, γ 4 | 2.8013 |
| Speedup, α 0.9, c 0.05, γ 4 | 3.4126 |
| Speedup, α 0.8, c 0.3, γ 4 | 1.528 |
| Speedup, α 0.8, c 0.6, γ 4 | 0.9887 |
| Best γ at α 0.8, c 0.05 | 8 |
| Best γ at α 0.8, c 0.3 | 3 |
The assumption of independent acceptance is a simplification: real acceptance varies by position and by text, so treat these as a model of the trade-off, not a forecast.1
Text description of the diagram
- At an acceptance rate of 0.3 the modelled speedup is 1.1876 times.
- At an acceptance rate of 0.9 the modelled speedup is 3.4126 times.
Trade-offs and edge cases
Speculative decoding is not free speed. The three levers are the acceptance rate, the draft cost and the number of drafts. A weak draft lowers the acceptance rate; a draft that is too big raises c until the cycle costs as much as it saves, as the chart below shows, where at c of 0.6 the speedup falls to 0.9887, below one.1 More drafts help until the chance that the whole run survives shrinks: at c of 0.05 the best γ is 8, at c of 0.3 it is 3.1
Text description of the diagram
- A draft that is too big raises c until the cycle costs as much as it saves.
- At a draft cost of 0.6 the modelled speedup is 0.9887 times, below one.
Other edge cases a strong answer names. First, serving at a large batch size: speculation works because the target pass has spare arithmetic capacity while waiting on memory. When continuous batching already keeps the graphics processing unit (GPU) busy with many requests, that spare capacity shrinks and so does the gain; measure on your own traffic. Second, memory: two models mean two sets of weights and a second cache; paged attention and the GPU fundamentals page explain the budget. Third, exactness: exactness holds for the rejection rule above. Methods that relax acceptance to speed up further, and floating-point differences between a batched pass and a single pass, can change outputs slightly, so check the method you deploy.
Medusa, EAGLE, self-speculative and n-gram drafting
Several variants avoid a separate draft model. Medusa adds extra decoding heads to the target model that predict several following tokens in parallel; its authors (Cai et al., 2024) present this as a way around acquiring and maintaining a separate draft model. Extrapolation Algorithm for Greater Language-model Efficiency (EAGLE) (Li et al., 2024) is a method whose title frames drafting as a problem of feature uncertainty: it drafts at the level of the target model's internal features instead of tokens.
Self-speculative decoding lets the target draft with itself; Zhang et al. (2023) call their version Draft & Verify, with no separate draft model. N-gram or prompt-lookup drafting copies a span that already appeared in the prompt, which helps when the output repeats the input, as in editing or summarizing.
These variants change where the drafts come from, not the verify step. Some use a relaxed acceptance test that is no longer exact, so read each method's paper for its acceptance rule and reported speedup before relying on it.
How to answer it in an interview
What is speculative decoding and why does it speed up inference?
A small draft model proposes a few tokens, and the large target model scores all of them in one forward pass. That is cheap because decoding is memory-bound, so checking several tokens costs about one pass. I accept each draft token with probability min of one and target over draft probability, and at the first rejection I resample from the normalized positive part of target minus draft, so the output distribution is exactly the target's. The gain depends on the acceptance rate and the draft cost, and it shrinks at large batch sizes.
The spoken structure: say what the two models do, say why the check is cheap, say why it is exact, then give the cost model and when it fails.
Common mistakes
- "Speculative decoding makes the answers a bit worse in exchange for speed." (wrong) With the accept-or-reject rule the distribution is unchanged. (right) "It is lossless in distribution; it trades extra compute for lower latency."
- "The draft model must be nearly as good as the target." (wrong) It only needs to agree often. (right) "A weak draft costs speed through a low acceptance rate, not correctness."
- "On rejection you just take the target's best token." (wrong) That biases the output. (right) "I resample from the normalized residual, the positive part of target minus draft."
- "It always speeds things up." (wrong) The gain depends on acceptance, draft cost and batch size. (right) "I measure it on the real workload; at a high draft cost the speedup can fall below one."
Follow-up questions
Does speculative decoding reduce accuracy?
No, not in its standard form: the rejection rule keeps the output distribution identical to the target model's, which both Leviathan et al. and Chen et al. state. Variants that relax the acceptance test, and small numeric differences, can change outputs, so check the specific method.
Is speculative decoding still used?
Yes, it remains an active topic as of October 2026: the vLLM project published a post titled "Exploring Speculative Decoding in vLLM on AMD GPUs" in August 2026, and variants such as Medusa and EAGLE (see the variants section above) build on the same draft-and-verify idea. Support differs by serving engine, draft method and model, so check the documentation of the engine version you run. It is a practical choice mainly when latency for one user matters more than raw throughput.
Who created speculative decoding?
Two groups published the exact sampling method with the accept-or-reject rule used here, independently. Leviathan, Kalman and Matias submitted "Fast Inference from Transformers via Speculative Decoding" on 2022-11-30. Chen and colleagues at DeepMind followed on 2023-02-02 and call it speculative sampling. The first paper was presented at the 2023 International Conference on Machine Learning (Proceedings of Machine Learning Research (PMLR), volume 202). Draft-and-verify decoding has precursors: blockwise parallel decoding by Stern, Shazeer and Uszkoreit (2018), and work by Xia and colleagues (2022) that used the name speculative decoding for sequence-to-sequence generation.
What is the acceptance rate in speculative decoding?
The acceptance rate is the average probability that the target model accepts a drafted token. In the worked example it is 0.8 for the toy distributions, and it drives the 2.8013 times speedup.1 Measure it per workload, since code and templated text are usually easier to draft than open-ended prose.
What is MTP speculative decoding, and is it the same as speculative decoding?
No, they are not the same. Multi-token prediction (MTP) is a training objective in which a model learns to predict more than one future token per position. Gloeckle et al. (2024) study training with multi-token prediction, and the DeepSeek-V3 technical report (2024) describes an MTP training objective in that model. Such extra predictions can serve as drafts that the same model then verifies, so MTP can supply the proposals, while speculative decoding is the draft-and-verify procedure. Check a model's technical report and release notes for how it uses MTP.
When would you choose speculative decoding over quantization?
Choose speculative decoding when you must keep the exact output distribution and have spare compute per request; quantization shrinks the bytes read per token but changes the weights. They can be combined, and a strong answer says to measure both on the real workload.
- Speculative decoding has a draft model propose tokens and the target model verify them in one pass.
- The check is cheap because decoding is limited by memory, not arithmetic.
- Accept with probability min(1, p/q) and resample rejections from the normalized residual to keep the target distribution exactly.
- Speedup depends on the acceptance rate, the draft cost and the number of drafts, and it shrinks at large batch sizes.
- Both lossless in distribution and not always faster: measure on your own workload.
Sources
- Fast Inference from Transformers via Speculative Decoding (arXiv 2211.17192v2) — arXiv (Leviathan, Kalman, Matias), 18 May 2023
- Accelerating Large Language Model Decoding with Speculative Sampling (arXiv 2302.01318v1) — arXiv (Chen et al., DeepMind), 2 Feb 2023
- Fast Inference from Transformers via Speculative Decoding, Proceedings of the 40th International Conference on Machine Learning, PMLR 202 — PMLR (ICML 2023), 23 Jul 2023
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (arXiv 2401.10774) — arXiv (Cai et al.), 19 Jan 2024
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (arXiv 2401.15077) — arXiv (Li et al.), 26 Jan 2024
- Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding (arXiv 2309.08168) — arXiv (Zhang et al.), 15 Sep 2023
- Better & Faster Large Language Models via Multi-token Prediction (arXiv 2404.19737) — arXiv (Gloeckle et al.), 30 Apr 2024
- DeepSeek-V3 Technical Report (arXiv 2412.19437) — arXiv (DeepSeek-AI), 27 Dec 2024
- Blockwise Parallel Decoding for Deep Autoregressive Models (arXiv 1811.03115) — arXiv (Stern, Shazeer, Uszkoreit), 7 Nov 2018
- Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (arXiv 2203.16487) — arXiv (Xia et al.), 30 Mar 2022
- Exploring Speculative Decoding in vLLM on AMD GPUs — vLLM project blog, 23 Aug 2026
