LLM inference explained: from prompt to generated tokens

What LLM inference actually does

LLM inference is running a trained large language model on new input to produce an output. In the decoder-only text generation covered here, the model repeatedly predicts a next token from the available context. Its learned weights stay fixed during the request; the evolving conversation is input, not a weight update.

A token is a vocabulary entry: it might represent a word, part of a word, punctuation, or another text fragment. The model returns scores over possible next tokens. A decoding rule selects one, appends it to the sequence, and repeats until a stopping condition or output limit is reached. Greedy selection and probabilistic sampling are different decoding choices. Hugging Face’s generation guide documents these controls.

This distinction matters when debugging. A fluent answer can still be factually wrong. Faster generation improves response time; it does not establish that the answer satisfies your task. Keep a small set of representative prompts with expected properties alongside your performance workload.

Follow a request from the API to the model

{
  "messages": [
    {"role": "system", "content": "Explain concepts to a new developer."},
    {"role": "user", "content": "Why does a longer prompt use more memory?"}
  ]
}

The server validates the input and formats messages using the model’s chat template. Roles and message boundaries become a token sequence with model-specific control tokens. Two models can expect different formats for the same messages. Use the tokenizer and template associated with the exact model revision; a formatting mismatch can hurt answers even when the server returns HTTP 200. Chat template documentation.

The request may wait while the scheduler makes room. The model then processes the input, produces output tokens, and the server converts them back to text. An API response can contain text chunks that do not correspond one-to-one with model tokens. Measure the stage you are trying to improve: a slow browser stream and a slow GPU step can have different causes.

Prefill reads the prompt; decode extends it

Prefill processes the prompt’s tokens and prepares the state needed for generation. Within a transformer layer, many prompt positions can be processed together while causal attention prevents a position from reading future positions. The final prompt position produces scores used to select the first output token.

Decode feeds the selected token back through the model to predict the next one. A key-value cache retains attention state from previously processed tokens. In the ordinary autoregressive loop, output positions depend on earlier choices, so a single request’s continuation cannot simply be computed like an already-known prompt. NVIDIA’s discussion of prefill and decode explains the different workloads.

Consider two illustrative requests: one has a 6,000-token document and asks for a 30-token label; another has a 60-token instruction and asks for an 800-token explanation. The first emphasizes prompt processing. The second spends more of its lifetime generating a continuation. They can prefer different scheduling choices even on the same model and GPU.

Measure waiting time and generation speed separately

For an illustrative request with 101 output tokens, a first token at 0.4 seconds, and the final token at 2.4 seconds, TPOT is (2.4 - 0.4) / (101 - 1) = 0.02 seconds, or 20 milliseconds. This arithmetic is an example, not a hardware benchmark. vLLM’s metric definitions explain why request-level TPOT and averages over streamed events can differ.

A service producing more total tokens per second can still feel slower to an individual user. Report median and tail latency, successful request count, and errors alongside throughput. In particular, do not hide rejected requests or timeouts by reporting only the successful requests’ speed.

Memory and scheduling determine how much work fits

Model weights are only one memory expense. For a simple estimate, eight billion parameters stored at two bytes each occupy 16 billion bytes, about 14.9 GiB. That leaves out the KV cache, temporary tensors, runtime allocations, and other overhead. A model fitting into memory is therefore a weaker condition than a useful production workload fitting.

Quantization uses lower-precision representations to reduce storage and sometimes computation costs. The supported formats and speedups depend on the hardware and implementation; reduced numerical precision also needs a quality check. Weight quantization and KV-cache quantization are separate decisions. Hugging Face’s quantization overview.

Continuous batching lets a serving system add new requests as existing ones finish, rather than waiting for an entire fixed batch to complete. This improves opportunities to share GPU work across requests of different lengths. The scheduler must still balance available memory and latency. Increasing admitted work without limits can increase waiting time or exhaust cache capacity. Continuous batching documentation.

Repeated prompt prefixes create another opportunity: reuse their already-computed attention state. Prefix caching reduces repeated prefill work when compatible tokens and model state match; it does not remove the work of generating a new answer. A benchmark that repeats one identical prompt may therefore differ substantially from varied production traffic. vLLM’s prefix caching guide.

Run a small experiment you can explain

1. Choose one model revision, tokenizer, serving version, and hardware configuration. Record the complete launch command and generation settings.
2. Test short and long prompts separately, with a fixed output cap. Record actual input and output token counts; a cap does not guarantee that the model generates that many tokens.
3. Begin with one active request, then increase concurrency gradually. Keep the prompt set and timing method stable.
4. Record TTFT, TPOT, total throughput, memory usage, failures, and output quality. Separate warm-up from the measured workload and identify whether prefixes were already cached.
5. Change one setting based on the observed bottleneck, repeat the workload, and retain the raw results. Stop increasing load when latency or reliability violates your application’s requirements.

If only long prompts have poor first-token latency, investigate prefill and queueing. If streaming slows as contexts grow, inspect cache pressure and attention work. If every request stalls under load, check admission, queue depth, and server health before changing the model. These are hypotheses to test, not diagnoses from a single chart.