LLM inference optimization: a practical guide to lower latency

Choose the latency target before the technique

For streaming chat, measure time to first token (TTFT) and the gaps between later tokens. For a background extraction job, measure the time until the complete, validated result arrives. A fast first token does not finish a JSON object or make an incomplete answer usable.

TTFT includes queueing, prompt processing, and transport. Time per output token (TPOT) averages generation time after the first token. NVIDIA defines the latter as (end-to-end latency - TTFT) / (output tokens - 1). Check that your measurement tool uses the same boundaries. NVIDIA latency metrics.

Write an acceptance rule such as: at the expected arrival rate, p95 TTFT stays below 800 ms, p95 TPOT stays below 60 ms, and task accuracy does not fall. These targets are illustrative. Choose yours from product requirements, and keep request lengths and traffic classes visible. The inference metrics guide covers collection and percentiles.

Match the symptom to the first experiment

Do not infer the bottleneck from a GPU utilization percentage alone. Compare idle and loaded runs, then inspect queue time, execution time, and memory behavior. The prefill and decode guide explains the phases behind these symptoms.

Calculate the maximum payoff before tuning

Consider one illustrative request: 0.4 s waiting, 2.4 s processing its prompt, 0.2 s of other startup work, and 201 output tokens. At an average 50 ms gap after the first token, generation adds 200 × 0.050 = 10 s. Total latency is 3 + 10 = 13 s.

Halving prompt processing saves 1.2 s, bringing completion to 11.8 s. Halving generation gaps saves 5 s, bringing completion to 8 s. Even eliminating prompt processing entirely only reaches 10.6 s. For this request, a large prefill improvement has a limited effect on completion time, although it improves responsiveness.

def latency(prompt_seconds, gap_seconds, output_tokens):
    if prompt_seconds < 0 or gap_seconds < 0 or output_tokens < 1:
        raise ValueError("Use nonnegative times and at least one output token")
    queue_seconds = 0.4
    other_seconds = 0.2
    return queue_seconds + prompt_seconds + other_seconds + (output_tokens - 1) * gap_seconds


cases = [
    ("Baseline", 2.4, 0.050, 201),
    ("Half prompt time", 1.2, 0.050, 201),
    ("Half generation gap", 2.4, 0.025, 201),
    ("101 output tokens", 2.4, 0.050, 101),
]
for label, prompt, gap, tokens in cases:
    print(f"{label}: {latency(prompt, gap, tokens):.2f} s")

Baseline: 13.00 s
Half prompt time: 11.80 s
Half generation gap: 8.00 s
101 output tokens: 8.00 s

Change the inputs to match a recorded request. The calculator holds queueing and token gaps fixed; it predicts a budget, not server performance. Under load, an execution change can also change waiting time. Do not add p95 stage timings and label the result p95 total latency: compute each request’s total first.

Reduce prompt work and test cache reuse

Inspect the rendered prompt before changing the server. Look for duplicated instructions, irrelevant retrieved passages, and conversation history that the task no longer needs. Compare answers against the full-context baseline; shorter prompts are useful only when the required evidence survives.

Prefix caching reuses previously computed key-value (KV) attention state for a compatible shared prefix. It reduces prefill work, while generating new output still requires decode. vLLM documents both the repeated-document use case and this limit. vLLM v0.30.0 prefix caching.

Test a cold request and repeated requests separately. Record how much prefix work was actually reused, rather than assuming that similar-looking documents share tokens. Review tokenization and cache routing with the prefix caching guide. Include a cold-cache run after each deployment change.

Optimize generation without losing useful output

First check whether the task needs the answer length you request. Compare a concise response specification with the existing prompt. Reducing a token ceiling has no effect when answers already stop below it; a lower ceiling can also truncate a required result. Score completeness alongside latency.

At small batches, decode often spends substantial time moving weights and attention state. Quantization reduces weight storage, but actual speed depends on the execution path and hardware. NVIDIA explains the memory constraint; AWQ is one primary example of low-bit weight quantization. NVIDIA inference optimization, AWQ paper.

Speculative decoding proposes several tokens and verifies them with the target model. The original algorithm preserves the target distribution through its verification and sampling procedure. Treat acceleration as something to measure on your prompts, rather than a fixed multiplier. Speculative decoding paper.

Compare quantization and speculative decoding as separate experiments. Save memory usage, low-load TPOT, and task scores. Under load, count requests per second that meet your latency targets, called goodput. A draft model adds work, so include its resource usage in the cost comparison.

Handle queueing, scheduling, and cache pressure

Continuous batching admits new work as other requests finish. It can improve utilization, but aggregate throughput alone does not establish acceptable user latency. NVIDIA describes how in-flight batching handles requests with different output lengths. NVIDIA serving techniques.

Sweep offered arrival rate while keeping prompt and output distributions fixed. At each rate, record completions, errors, queue time, and latency percentiles. If the pending queue grows throughout a run, extend the observation or lower the offered load before claiming sustainable capacity.

For streaming stalls, test chunked prefill under mixed traffic. Smaller prefill allocations can protect token gaps while delaying prompt completion. For insufficient KV capacity, vLLM can preempt and later recompute requests. Its v0.30.0 tuning guide describes both mechanisms and warns of synchronization costs when adding tensor parallelism. vLLM v0.30.0 tuning.

Use the continuous batching guide for scheduling details and the GPU memory worksheet for capacity. Compare a smaller active set against more cache memory before scaling out. For expert-based models, also investigate the placement and traffic issues in mixture-of-experts inference.

Accept a change with a repeatable comparison

1. Freeze the baseline. Save model revision, engine version, hardware, rendered prompts, sampling settings, and arrival schedule.
2. State the prediction. For example: shared-prefix requests should start sooner, while cold requests should show little benefit.
3. Change one variable. Repeat baseline and candidate runs, alternate their order, and separate warmup from measurement.
4. Count failures. Include timeouts, rejected requests, invalid structured outputs, and quality failures. Report success latency alongside the failure rate.
5. Price usable work. Divide total serving cost by completed requests that met the latency and quality rules. Use the same workload and observation window.

Suppose two illustrative hourly deployments cost $2 and $3. If they deliver 2,000 and 4,000 acceptable requests, their costs are $0.001 and $0.00075 per acceptable request. The more expensive deployment delivers cheaper usable work. Include any additional replicas or draft-model resources in both totals.

Reject a comparison if the candidate silently serves shorter outputs, a warmer cache, or fewer successful requests. Start from the vLLM tutorial, then use the vLLM course or SGLang course to investigate engine behavior.

LLM inference optimization FAQ

What should I optimize first? Measure the failing request class at the expected load. Start with queueing if waiting dominates, prompt work if startup dominates, or generation if long answers dominate completion time.

Should I add more GPUs? Compare the reason for adding them: fitting weights, increasing active cache capacity, or serving more requests. Include communication and total cost. The parallelism guide explains placement choices.

How do I know an optimization worked? Repeat the same workload and check the original latency, failure, quality, and cost rules together. A faster median with worse tail latency is a tradeoff to evaluate, not automatic success.