LLM inference metrics: TTFT, TPOT, ITL, and throughput

LLM inference metrics at a glance

Latency metrics describe one user’s experience. Throughput metrics describe how much work the whole server does. You need both: a server can post high throughput by batching aggressively while every user waits a long time for their first token.

Time to first token (TTFT)

TTFT covers everything before the first token arrives: network time, waiting in the server’s queue, and the prefill pass over the prompt. It grows with prompt length and with load, because requests wait longer for a slot. For chat interfaces it is usually the most important number, since it decides how responsive the product feels.

When TTFT is high, first check queue time. vLLM reports it separately as vllm:request_queue_time_seconds, next to vllm:time_to_first_token_seconds. vLLM: metrics. A long queue points to too little capacity; a short queue with slow TTFT points to long prompts or slow prefill. LLM inference explained describes the prefill phase.

TPOT vs inter-token latency (ITL)

After the first token, the answer streams during the decode phase. TPOT summarizes a request’s streaming speed as one average, and ITL keeps every gap between tokens. NVIDIA’s benchmarking guide defines the per-request average as (end_to_end_latency - TTFT) / (output_tokens - 1), which excludes the first token because it belongs to TTFT. NVIDIA: LLM inference metrics.

The two can tell different stories. A request with a smooth 50 ms rhythm and one with mostly 30 ms gaps plus a 500 ms pause can have similar TPOT, but users notice the pause. Pauses often happen when a long prompt from another user is prefilled in the same step. Continuous batching explained shows how chunked prefill reduces them.

Calculate the metrics from timestamps

All the latency metrics come from two things: when the request was sent and when each token arrived. This script calculates them for one request, then summarizes ten illustrative requests with percentiles and goodput.

import math


def request_metrics(sent, token_times):
    """Timestamps in seconds; token_times holds one entry per output token."""
    ttft = token_times[0] - sent
    e2e = token_times[-1] - sent
    gaps = [later - earlier for earlier, later in zip(token_times, token_times[1:])]
    tpot = (e2e - ttft) / (len(token_times) - 1)
    return {"ttft": ttft, "e2e": e2e, "tpot": tpot, "itl": gaps}


def percentile(values, pct):
    """Nearest-rank percentile: the smallest value covering pct percent of samples."""
    ordered = sorted(values)
    return ordered[max(0, math.ceil(pct / 100 * len(ordered)) - 1)]


one = request_metrics(0.0, [0.42, 0.47, 0.53, 0.58, 0.66, 0.71])
print(f"TTFT {one['ttft'] * 1000:.0f} ms, end to end {one['e2e'] * 1000:.0f} ms, "
      f"TPOT {one['tpot'] * 1000:.0f} ms")
print("ITL gaps (ms):", [round(gap * 1000) for gap in one["itl"]])

# Illustrative results for ten requests: (TTFT seconds, TPOT seconds).
results = [(0.31, 0.041), (0.35, 0.044), (0.29, 0.040), (0.42, 0.047), (0.38, 0.043),
           (0.33, 0.052), (1.90, 0.045), (0.36, 0.049), (0.40, 0.071), (0.34, 0.046)]
ttfts = [ttft for ttft, _ in results]
print(f"TTFT mean {sum(ttfts) / len(ttfts):.2f} s, p50 {percentile(ttfts, 50):.2f} s, "
      f"p90 {percentile(ttfts, 90):.2f} s")

slo = {"ttft": 0.5, "tpot": 0.050}
good = [r for r in results if r[0] <= slo["ttft"] and r[1] <= slo["tpot"]]
window_seconds = 4.0
print(f"{len(good)} of {len(results)} requests met both targets: "
      f"goodput {len(good) / window_seconds:.2f} req/s vs "
      f"throughput {len(results) / window_seconds:.2f} req/s")

TTFT 420 ms, end to end 710 ms, TPOT 58 ms
ITL gaps (ms): [50, 60, 50, 80, 50]
TTFT mean 0.51 s, p50 0.35 s, p90 0.42 s
7 of 10 requests met both targets: goodput 1.75 req/s vs throughput 2.50 req/s

Notice the mean TTFT of 0.51 seconds. It is higher than the 90th percentile, because one request waited 1.9 seconds. A mean hides that outlier and also misrepresents the typical request. Report the median, the 90th or 99th percentile, and the maximum instead. These timings are illustrative, not measurements of a real server.

Throughput vs goodput

Output throughput is total output tokens divided by the time from the first request to the last response. Request throughput is completed requests per second. Both rise with batch size, so on their own they reward a server that makes everyone wait.

Goodput counts only requests that met your latency targets, such as TTFT under 500 ms and TPOT under 50 ms. In the example above the server completes 2.5 requests per second, but only 1.75 meet both targets. DistServe popularized goodput as the measure to optimize for LLM serving, because it ties capacity to the experience users actually get.

Also watch per-user tokens per second, one request’s output length divided by its end-to-end latency. It approaches 1 / TPOT for long answers and is a useful check that heavy batching has not made individual streams too slow to read.

Measure TTFT and TPOT on your own server

This client sends one streaming chat request to any OpenAI-compatible endpoint, such as vLLM or SGLang, and times it. It uses only the Python standard library. It asks the server for a final usage report, because streamed text chunks do not always correspond one to one with tokens.

import json
import os
import time
import urllib.request

BASE_URL = os.environ.get("BASE_URL", "http://127.0.0.1:8000")
MODEL = os.environ["MODEL"]
API_KEY = os.environ.get("API_KEY", "")

body = {
    "model": MODEL,
    "messages": [{"role": "user", "content": "Explain the KV cache in three sentences."}],
    "max_tokens": 128,
    "temperature": 0,
    "stream": True,
    "stream_options": {"include_usage": True},
}
headers = {"Content-Type": "application/json"}
if API_KEY:
    headers["Authorization"] = f"Bearer {API_KEY}"
request = urllib.request.Request(
    f"{BASE_URL}/v1/chat/completions", data=json.dumps(body).encode(), headers=headers
)

sent = time.perf_counter()
chunk_times, output_tokens = [], None
with urllib.request.urlopen(request, timeout=300) as response:
    for raw_line in response:
        line = raw_line.decode().strip()
        if not line.startswith("data: "):
            continue
        data = line[len("data: "):]
        if data == "[DONE]":
            break
        event = json.loads(data)
        if event.get("usage"):
            output_tokens = event["usage"]["completion_tokens"]
        choices = event.get("choices") or []
        if choices and choices[0].get("delta", {}).get("content"):
            chunk_times.append(time.perf_counter())

if not chunk_times:
    raise SystemExit("The server returned no text.")
ttft = chunk_times[0] - sent
e2e = chunk_times[-1] - sent
tokens = output_tokens or len(chunk_times)
print(f"TTFT {ttft * 1000:.0f} ms, end to end {e2e * 1000:.0f} ms, {tokens} output tokens")
if tokens > 1:
    print(f"TPOT {(e2e - ttft) / (tokens - 1) * 1000:.1f} ms")

A single request measures an idle server. Run it a few times to warm up, then measure under load. If you do not have a server yet, the vLLM tutorial sets one up on port 8000 with an API key.

Benchmark under load with percentiles

For load tests, use a benchmark tool rather than a hand-written loop. vLLM ships vllm bench serve, which sends many requests, reports TTFT, TPOT, ITL, and end-to-end latency at the percentiles you choose, and can compute goodput from targets given in milliseconds. vLLM: bench serve.

vllm bench serve \
  --backend openai-chat \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/chat/completions \
  --model "$MODEL" \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 256 \
  --num-prompts 200 \
  --max-concurrency 16 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,99 \
  --goodput ttft:500 tpot:50 \
  --save-result

• Fix the load shape. --max-concurrency holds a set number of requests in flight, and --request-rate sends requests at a set rate. Test several levels and plot latency against throughput instead of reporting one point.
• Match your traffic. Random prompts are convenient, but prompt and output lengths change every metric. Use lengths that resemble production, or a dataset of real prompts.
• Warm up and repeat. Discard the first run, repeat each level, and keep the saved results with the server version, model revision, and GPU.
• Compare fairly. Use the same client, prompts, and targets for every server you test. vLLM vs SGLang walks through a fair comparison.

LLM inference metrics FAQ

What is a good TTFT? It depends on the product. Interactive chat usually aims for well under a second at the 90th percentile; offline batch jobs may not care. Set targets from user expectations, then measure goodput against them.

Is TPOT the same as ITL? They are related. TPOT is one average per request; ITL is the list of individual gaps. Some tools use the names differently, so check definitions.

Why is my throughput high but users complain? Throughput rewards large batches, which can raise TTFT and TPOT. Look at high percentiles of latency and at goodput instead.

Which techniques improve which metric? Speculative decoding mainly improves TPOT at low load. Quantization frees memory for larger batches and can speed up decode. Chunked prefill trades a little TTFT for smoother ITL.