vLLM vs SGLang: a measured comparison and a fair benchmark plan

Compare a deployment, not an engine name

Imagine two services using the same language model. One answers short chat messages; the other reads long source files before producing a few lines of code. Their bottlenecks differ. Processing the prompt is called prefill. Producing the answer token by token is decode. A configuration that improves one phase may delay the other.

vLLM and SGLang both serve language models and manage requests sharing accelerator resources. Begin with a practical compatibility check: your exact model architecture, quantization format, hardware, context length, and required API behavior. A fast configuration that cannot produce your application’s required output is not a candidate.

Published measurements: tuning changes the result

SGLang contributor zhyncs published the following experiment on February 10, 2025: H200 hardware, Qwen/Qwen2.5-Coder-7B-Instruct, synthetic 30,000-token inputs, 500-token outputs, and one arriving request per second. Values below are transcribed from the original benchmark logs.

Larger SGLang prefill chunks raise reported throughput, but worsen p99 gaps between tokens. The highest throughput configuration therefore does not win every latency metric.

Changing the workload changes the ranking

Another historical example comes from the vLLM team’s September 2024 comparison. For Llama 3 8B on one H100, its charts place vLLM 0.6.0 ahead of SGLang 0.3.0 on ShareGPT and decode-heavy requests, while SGLang leads on prefill-heavy requests. vLLM used ten scheduler steps; competitors used default configurations. This was a throughput test with requests arriving together, not a production latency guarantee.

The accompanying reproduction instructions acknowledge that output-token counts did not align exactly across engines. That detail matters: finishing shorter answers can inflate requests per second. Preserve both completed-request counts and actual generated-token counts when comparing results.

Do not combine these experiments into a leaderboard. Their models, versions, hardware, request patterns, and measurement boundaries differ. Use them to formulate testable questions: Does longer prefill block streaming? Does the benefit survive lower concurrency? Does the client count generated tokens the same way for both servers?

Measure prefix reuse explicitly

A repeated system prompt or shared document can reuse previously computed key/value tensors. SGLang’s RadixAttention design organizes reusable prefixes in a radix tree. vLLM also supports automatic prefix caching. Treating prefix reuse as exclusive to SGLang would make the comparison misleading.

For conventional autoregressive attention, prefix reuse saves prompt processing; it does not eliminate the work of generating a new answer. Measure two separate conditions: independent prompts with caching disabled, and realistic repeated prefixes with caching enabled. Record how caches are warmed or cleared before each trial. Replaying the same prompts repeatedly without tracking cache state can turn a supposed general benchmark into a cache-hit benchmark.

Decide what a useful response costs

Set latency limits before running the comparison. For example, an application might require 95% of answers to start within a chosen time budget. That is your product requirement, not a universal target. Report failure rates alongside latency: excluding timed-out requests can make an overloaded server look faster.

For a cost comparison, divide the actual infrastructure cost of a trial by the number of successful answers meeting both quality and latency requirements. Include every GPU, required CPU resources, and any draft model. Do not infer production savings from peak tokens per second alone; idle capacity and traffic bursts also affect the bill.

Control the experiment before tuning

1. Freeze the model and tokenizer revisions, chat template, prompt corpus, precision, output limit, and stopping policy. Save the engine versions, container digests, driver version, GPU count, and launch commands.
2. Run one engine at a time on the same otherwise idle machine. Give both equivalent memory and parallelism budgets. Check actual memory use; similarly named percentage flags need not reserve the same resources.
3. Warm up compilation and kernels before measurement. Keep warmup outside the timed interval. Disable caching for an independent-prompt baseline, then test production cache behavior separately.
4. Sweep offered request rates and prompt/output lengths. Repeat each condition at least three times, alternate engine order, and publish the spread rather than only the fastest run.
5. Validate application behavior: structured-output validity, tool arguments, answer quality, cancellation, and timeout handling. Compare performance only after both candidates pass the same checks.

Keep two result sets: documented defaults and configurations tuned with the same effort. A default comparison estimates setup experience. A tuned comparison estimates what your team can operate. Mixing one tuned server with the other’s defaults obscures that distinction.

Use one client against both servers

Use one fixed client build and the same endpoint protocol for both engines. The SGLang benchmark client supports both OpenAI-compatible servers and can save per-request details. The example below assumes a prepared Python 3.10+ environment with the SGLang 0.5.3 client installed and a reachable server exposing the tutorial-model alias. Set OPENAI_API_KEY to that server’s credential. Run the servers sequentially on the same GPU; this is a client recipe, not a server installer.

#!/usr/bin/env bash
set -euo pipefail
engine_label="${1:?Use: bash compare.sh vllm 8000}"
server_port="${2:?Provide the server port}"
mkdir -p benchmark-results
python -m pip freeze > benchmark-results/client-requirements.txt
python - <<'PYTHON'
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="Qwen/Qwen3-0.6B",
    revision="c1899de289a04d12100db370d81485cdf75e47ca",
    allow_patterns=["*.json", "*.txt"],
    local_dir="benchmark-tokenizer",
)
PYTHON

for request_rate in 1 2 4; do
  python -m sglang.bench_serving \
    --backend vllm \
    --base-url "http://127.0.0.1:${server_port}" \
    --model tutorial-model --tokenizer ./benchmark-tokenizer \
    --dataset-name random-ids \
    --random-input-len 1024 --random-output-len 128 \
    --random-range-ratio 1 --seed 42 \
    --num-prompts 300 --request-rate "${request_rate}" \
    --warmup-requests 10 --output-details \
    --output-file "benchmark-results/${engine_label}-r${request_rate}.jsonl"
done

Use bash compare.sh vllm 8000, stop that server, start SGLang, then use bash compare.sh sglang 30000. Here --backend vllm selects the common completions protocol for both. Keep the saved client environment unchanged. Match the server alias on both engines. Check installed flags with python -m sglang.bench_serving --help before starting.

For server setup, see our vLLM tutorial and the SGLang quickstart. When continuing from the tutorial, use export OPENAI_API_KEY="$VLLM_API_KEY". Pin both servers to model revision c1899de289a04d12100db370d81485cdf75e47ca, the same tokenizer, and the tutorial’s float16 precision. Disable prefix caching for this baseline. Verify requested and actual token lengths in the saved output; text tokenization and stop handling can differ. Random token IDs avoid a changing external prompt dataset, but their decoded text can be gibberish; this tests capacity, not answer quality. Follow it with representative application prompts and a separate quality evaluation. Stop the server processes and release the GPU allocation when finished.

Choose the result your team can sustain

Choose vLLM if its measured deployment meets your requirements with lower cost or operational effort. Choose SGLang if its measured deployment does better under those same requirements. If the difference is within run-to-run variation, prioritize integration quality, debugging, upgrade reliability, and the team’s ability to operate the service.

Keep the winning commands, environment records, raw client output, and quality checks together. Re-run them when changing a model, engine release, GPU, or major traffic pattern. That gives you a defensible decision and a regression baseline instead of a conclusion tied to somebody else’s workload.