Prefill vs. decode: how LLM inference latency works

What is the difference between prefill and decode?

Prefill processes the prompt and builds the attention state needed to continue it. Decode repeatedly extends that sequence. In ordinary autoregressive generation, the prefill forward pass produces the logits used to select the first output token; subsequent passes produce the remaining tokens. The distinction is about model execution, even if an application displays everything only after generation finishes. NVIDIA inference overview.

Use this distinction to frame a specific question: is the service slow to start, slow to continue, or both? A report that “the model takes ten seconds” does not identify either phase. The LLM inference introduction covers the surrounding request lifecycle.

The KV cache connects the two phases

A causal transformer stores attention keys and values for processed tokens in a KV cache. Decode reuses that state instead of rebuilding earlier keys and values on every step, then adds state for the newly processed token. It still performs attention over the relevant cached context. Caching removes repeated work; it does not make an arbitrarily long context free. Hugging Face cache explanation.

For example, asking for a two-sentence summary of a long document can require substantial prompt processing but little generation. Asking for a long report from a short instruction reverses that balance. Keep both lengths in your benchmark dataset. Our KV cache guide explains the memory calculation, while prefix caching covers reusing compatible prompt state across requests.

How prefill and decode affect TTFT and TPOT

Time to first token (TTFT) is the interval from request submission to the first received output token. It includes more than prefill: queueing, transport, and other request processing also contribute. Time per output token (TPOT) summarizes the interval after the first token. For a request with N output tokens, use (last_token_time - first_token_time) / (N - 1). NVIDIA documents this average under inter-token latency and warns that tool definitions vary. NVIDIA benchmarking metrics.

Record individual inter-token gaps when available, because an average can conceal a visible stall. For zero output tokens, TTFT is undefined; for one output token, there is no after-first-token TPOT. Exclude those cases from the corresponding aggregates instead of assigning zero. The inference metrics guide explains percentile reporting.

A worked example: which phase is worth optimizing?

Consider an invented request with a 4,000-token prompt and 101 output tokens. It waits 200 ms for admission, spends 300 ms in prompt processing, and uses another 100 ms in combined transport and other overhead before the first token arrives. The remaining 100 token intervals average 40 ms. These numbers illustrate the arithmetic; they are not measurements of a model or GPU.

TTFT = 200 ms + 300 ms + 100 ms = 600 ms
After-first-token time = 100 intervals × 40 ms = 4,000 ms
Time to last token = 600 ms + 4,000 ms = 4,600 ms
TPOT = (4,600 ms - 600 ms) / (101 - 1) = 40 ms

Halving only prompt processing saves 150 ms, making TTFT 450 ms and total time 4,450 ms. Reducing only TPOT to 30 ms saves 1,000 ms, making total time 3,600 ms while leaving TTFT unchanged. Removing the 200 ms queue delay improves the start more than halving prefill in this example.

The right choice depends on the product target. A chat interface might prioritize starting within half a second; a report generator might prioritize completion time. Choose the metric before choosing the technique, and repeat the calculation for short and long answers.

Is prefill compute-bound and decode memory-bound?

That is a useful starting hypothesis. A sufficiently large prefill exposes parallel work across known tokens. Decode at small batch sizes often spends much of its time moving model weights and cache data relative to the arithmetic performed. Batching lets multiple requests share weight reads. NVIDIA inference optimization.

Treat these as operating regimes, not permanent labels. Short prompts, larger decode batches, long contexts, model architecture, and communication across GPUs can change the limiting resource. A practical investigation should collect a profile at the failing load level. High GPU utilization alone does not establish that arithmetic throughput is the problem.

Before adding GPUs, check whether the constraint is model capacity, active cache capacity, latency, or aggregate throughput. Each needs a different comparison. Tensor parallelism vs pipeline parallelism explains the placement choices, and GPU memory for inference provides a capacity worksheet.

Chunked prefill: share execution time more carefully

Chunked prefill divides prompt processing into smaller pieces that can be scheduled alongside ongoing generation. In vLLM’s documented policy, decode work receives priority, and pending prefill uses the remaining token budget. Smaller prefill allocations can protect inter-token latency; larger allocations can finish waiting prompts sooner. vLLM v0.30.0 tuning guide.

Test this with mixed traffic: start several ongoing answers, then introduce long prompts. Record whether those arrivals create streaming stalls and whether prompt starts become slower after tuning. An isolated single-request benchmark cannot reveal this interaction. Continuous batching explains how active requests enter and leave a serving batch.

Disaggregated prefill and decode: use separate worker pools

Prefill-decode disaggregation places the phases on separate workers. The system transfers the prompt’s KV state to the decode side, allowing the two pools to scale independently. NVIDIA Dynamo documents this arrangement and cautions that short prompts, low concurrency, or a slow transfer path can favor an aggregated deployment. Dynamo disaggregated serving.

Evaluate the complete path: waiting for a prefill worker, executing prefill, moving usable state, admission to decode, and streaming output. Cache movement and coordination have costs. A faster prefill pool is insufficient if it delivers work faster than the decode pool can accept it. Include transfer latency and pool imbalance in the comparison.

Chunking changes scheduling within an execution resource; disaggregation changes where phases run. They address different decisions and can coexist. Begin with a working aggregated baseline and require a measurable benefit at your latency targets before adopting separate pools. The Dynamo course explores distributed serving architecture.

A practical checklist for diagnosing LLM latency

1. Capture the workload. Record model and server versions, prompt and output lengths, arrival rate, concurrent requests, and cache state. Separate startup from steady-state measurements.
2. Split the symptoms. Plot median and p95 TTFT alongside TPOT, streaming gaps, and time to last token. Keep request classes separate so a changing mix does not hide a regression.
3. Locate waiting time. Compare client observations with server queue and execution timings. If queueing dominates, reducing an isolated kernel’s time may miss the main cause.
4. Change one input dimension. Compare short and long prompts with matched outputs, then short and long outputs with matched prompts. Repeat at idle and realistic load.
5. Test one intervention. State the expected effect first: for example, reusable prefixes should reduce prompt work, while scheduling changes should alter interference between requests.
6. Check the whole outcome. Keep only changes that meet the relevant latency targets without unacceptable errors, answer-quality changes, or cost. Save the baseline and comparison together.

For a first hands-on baseline, follow the vLLM tutorial. Understanding the two phases turns a vague speed complaint into an experiment with a predicted result and a clear acceptance criterion.