Speculative decoding explained: faster LLM inference

What is speculative decoding?

Speculative decoding is an inference technique in which a cheap method proposes several next tokens and the full model, called the target model, checks all of them in a single forward pass. The target model keeps the longest run of proposals it agrees with, adds one token of its own, and discards the rest. The technique was introduced in Fast Inference from Transformers via Speculative Decoding and, independently, in Accelerating Large Language Model Decoding with Speculative Sampling.

Ordinary generation produces exactly one token per target forward pass, because each token depends on the one before it. Speculative decoding breaks that one-to-one link. When the proposals are good, each expensive pass yields several tokens, so the response streams faster. When they are poor, the pass still yields one token, so the worst case is close to normal decoding plus the cost of proposing.

If you are new to the prefill and decode phases, read LLM inference explained first. Speculative decoding speeds up the decode phase only.

Why checking several tokens costs about the same as generating one

At small batch sizes, a decode step spends most of its time reading model weights and the KV cache from GPU memory, not doing arithmetic. Scoring five positions instead of one reuses the same weights that were already loaded, so the extra work is mostly spare compute. That is why verification is cheap: it looks like a short prefill over the proposed tokens.

This also explains the main limitation. As the batch grows, the GPU does more useful arithmetic per weight it reads, and the spare compute disappears. Verifying tokens that will be rejected then takes capacity away from other requests. The vLLM guide describes the benefit as applying to medium-to-low request rates and memory-bound workloads. vLLM: speculative decoding.

How one draft and verify step works

1. The proposer suggests k tokens that could follow the current text. It might be a small draft model, a lookup in the prompt, or extra prediction heads attached to the target model.
2. The target model runs one forward pass over the current position plus the k proposals. This gives its own prediction at every proposed position.
3. Starting from the first proposal, each token is accepted while it matches what the target model would choose. With greedy decoding this is an exact comparison. With sampling, a rejection sampling rule accepts or rejects each token so that the output distribution matches the target model.
4. At the first rejection, the target model’s own token replaces it and the remaining proposals are discarded. If every proposal is accepted, the target model adds one bonus token from the same pass.
5. The KV cache entries for rejected positions are discarded, and the loop repeats from the new end of the text.

Estimate the speedup from the acceptance rate

The acceptance rate is the probability that a proposed token is accepted. If each proposal is accepted independently with probability a and the proposer suggests k tokens, the expected number of tokens produced per target pass is (1 - a^(k+1)) / (1 - a). The speculative decoding paper derives this formula and a matching wall clock estimate that also counts the proposer’s cost.

def expected_tokens(acceptance, draft_tokens):
    """Mean tokens produced per target forward pass (Leviathan et al., 2023)."""
    if acceptance == 1:
        return draft_tokens + 1
    return (1 - acceptance ** (draft_tokens + 1)) / (1 - acceptance)


def speedup(acceptance, draft_tokens, draft_cost):
    """Idealized wall-clock gain; draft_cost is one draft step / one target step."""
    return expected_tokens(acceptance, draft_tokens) / (draft_tokens * draft_cost + 1)


print("acceptance  k=2    k=4    k=8   (tokens per target pass)")
for acceptance in (0.5, 0.7, 0.9):
    row = [expected_tokens(acceptance, k) for k in (2, 4, 8)]
    print(f"{acceptance:>10}  " + "  ".join(f"{value:5.2f}" for value in row))

print()
print("idealized speedup with draft_cost = 0.05")
for acceptance in (0.5, 0.7, 0.9):
    row = [speedup(acceptance, k, 0.05) for k in (2, 4, 8)]
    print(f"{acceptance:>10}  " + "  ".join(f"{value:5.2f}x" for value in row))

assert abs(expected_tokens(0.7, 4) - 2.7731) < 1e-4

acceptance  k=2    k=4    k=8   (tokens per target pass)
       0.5   1.75   1.94   2.00
       0.7   2.19   2.77   3.20
       0.9   2.71   4.10   6.13

idealized speedup with draft_cost = 0.05
       0.5   1.59x   1.61x   1.43x
       0.7   1.99x   2.31x   2.28x
       0.9   2.46x   3.41x   4.38x

Two patterns stand out. First, acceptance rate matters more than proposal length: at 50 percent acceptance, proposing eight tokens instead of four barely changes the tokens per pass and lowers the estimated speedup, because the extra drafts are usually thrown away. Second, the gain depends on how cheap the proposer is. These are idealized calculations that assume independent acceptance and free verification. They are not benchmark results; real systems add scheduling, memory, and kernel overheads.

Draft models, n-grams, EAGLE, Medusa, and MTP compared

EAGLE is described in EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty and Medusa in Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. The DeepSeek-V3 technical report explains how multi-token prediction modules can serve speculative decoding. N-gram methods need no extra model at all, which makes them a low-risk first experiment.

Enable speculative decoding in vLLM and SGLang

vLLM takes a JSON speculative configuration. This n-gram example from the vLLM documentation proposes up to four tokens by matching two to five token patterns already in the context. Replace the placeholder with your model ID.

vllm serve <target-model> \
  --speculative-config '{
    "method": "ngram",
    "num_speculative_tokens": 4,
    "prompt_lookup_min": 2,
    "prompt_lookup_max": 5
  }'

The same option accepts a draft model, EAGLE, MTP, and other methods. See vLLM: n-gram speculation and vLLM: EAGLE draft models for the fields each method needs. SGLang uses separate launch flags. This EAGLE example comes from the SGLang documentation:

python3 -m sglang.launch_server \
    --model meta-llama/Llama-2-7b-chat-hf \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8 \
    --log-level warning

Option names change between releases, so copy flags from the documentation for the version you run. SGLang: speculative decoding lists the supported algorithms. To set up a server from scratch, start with the vLLM tutorial.

When speculative decoding makes things slower

• High concurrency. Large batches already use the GPU’s compute, so verifying rejected tokens costs real throughput. Measure at the request rates you actually serve.
• Low acceptance. Creative writing at a high temperature, or a draft model trained on different data, accepts few tokens. The proposer’s work is then mostly wasted.
• An expensive proposer. A draft model that is too large eats the savings. The calculation above shows how quickly the gain falls as draft_cost rises.
• Memory pressure. A draft model needs its own weights and KV cache. That memory is no longer available for batching more requests.

Treat speculative decoding as a latency optimization for interactive traffic, not a free throughput increase. It usually improves time per output token for individual users; whether it improves total tokens per second depends on load.

Measure the gain on your own workload

1. Pick prompts that represent your traffic, including their typical output lengths and sampling settings.
2. Run the same benchmark with and without speculation at several fixed concurrency levels, such as 1, 8, and 32.
3. Compare time per output token, time to first token, and total output tokens per second at each level. LLM inference metrics explains how to measure each one.
4. Record the acceptance rate. vLLM exposes acceptance counters, such as vllm:spec_decode_num_accepted_tokens_per_pos, on its Prometheus endpoint. vLLM: metrics.
5. With greedy decoding, check that responses match the baseline on a sample of prompts before rolling out.

A typical outcome is a clear gain at low concurrency that shrinks as load grows. If you serve both interactive and batch traffic, you may want speculation only on the latency-sensitive deployment.

Speculative decoding FAQ

Does speculative decoding change the model’s answers? No, not by design. The verification rule keeps the target model’s output distribution. Small differences can still appear from floating point effects, so test on your own prompts.

What is a good acceptance rate? It depends on the proposer and the task. Use the calculation above: once acceptance is known, you can see how many tokens per pass to expect and which proposal length makes sense.

Does it help prefill or time to first token? No. It speeds up the decode loop. Time to first token is dominated by queueing and prompt processing.

Can I combine it with quantization? Often, yes. Engine support for each combination varies, so check the documentation. LLM quantization explained covers the quantization side.