vLLM

Learn to run vLLM 0.28.0 in production: serve models from Python and an OpenAI-compatible API, see how its scheduler and paged KV cache work inside, and tune caching, structured output, quantization, and multi-GPU serving with measurements.

Beginner → production
13 units · approximately 14 hours

How to work through this course

Before you begin: The free Foundations course or equivalent knowledge of tokens, the KV cache, and batching, plus comfort with Python and a Linux shell. Hands-on steps need an NVIDIA GPU; every lesson also has an exercise that runs on a laptop.

By the end: Deploy a pinned vLLM service, explain where its memory and time go, and choose settings by measured latency, throughput, and quality on your own traffic.

Understand the idea. Read each section in order and study its diagrams. When a pause-and-predict question appears, commit to an answer before you reveal it. Look up unfamiliar terms in the glossary below.
Try it yourself. Run the code examples and compare your output with the lesson's. Use the browser labs where available; steps that need a GPU are optional and say so.
Check your understanding. Answer the knowledge check at the end of each lesson and make sure you can say why the other options are wrong. Complete all units to unlock the final quiz; the passing score is 80%.

The optional project brings the lessons together. Keep notes on your choices, results, and any failures so you can explain what you learned.

Course orientation

What is vLLM?

vLLM is an open-source inference engine for large language models. It loads a model's weights onto one or more GPUs, accepts requests from Python or an OpenAI-compatible HTTP server, decides which requests run at every step, keeps each request's KV cache in fixed-size blocks, and streams the generated tokens back.

The problem it solves

Serving many users from one GPU is mostly a scheduling and memory problem. Requests arrive at random, prompts range from ten tokens to a hundred thousand, and answers finish at different times. Batching whole requests together leaves the GPU idle while it waits for the longest one, and reserving cache space for the longest possible answer wastes most of the memory. vLLM schedules work one step at a time and hands out cache memory in small blocks as each request grows.

Request path

01API server and Python entry points
`vllm serve` exposes OpenAI-compatible endpoints such as `/v1/chat/completions`, and the `LLM` class runs batches in Python. Both render chat templates, tokenize prompts, and validate sampling settings.
02Engine core and scheduler
At every step the scheduler picks which waiting and running requests get tokens, within a per-step token budget and a limit on concurrent sequences, and splits long prompts into chunks.
03KV cache manager
GPU memory left after the weights becomes a pool of cache blocks of 16 tokens each. A block table maps each request's positions to blocks, and requests that share a prefix can share blocks.
04Model runner and workers
One worker per GPU runs the forward pass with optimized attention kernels, compiled graphs, and CUDA graphs, then samples the next token for every request in the batch.
05Output processing
Detokenization, stop checks, streaming, and metrics turn sampled token IDs into responses, and free a request's blocks when it finishes or is cancelled.

What it covers

Scheduling and batching requests on the GPUs of one server, including preemption when the KV cache runs out.
Managing KV cache memory: paging, prefix caching, and lower-precision cache formats.
Running the model fast: compiled graphs, quantized weights, speculative decoding, and tensor, pipeline, data, and expert parallelism.
Serving the OpenAI-compatible API, structured output, tool-call parsing, and Prometheus metrics.

Outside its scope

Training or fine-tuning models. vLLM serves weights that already exist, including LoRA adapters trained elsewhere.
Routing across a fleet of servers, autoscaling, and cache-aware load balancing. Systems such as NVIDIA Dynamo sit above vLLM for that.
Production access control. `--api-key` protects only part of the API, so authentication, rate limits, and logging belong in a gateway in front of it.
Guaranteed support for every model and GPU. Features depend on the model architecture, the hardware, and the release you pin.

Key terms in this course

PagedAttention: vLLM's attention scheme that reads the KV cache from fixed-size blocks through a block table, so a request's cache need not be contiguous.
KV block: A fixed-size slice of the KV cache, 16 token positions by default in vLLM 0.28.0.
Block table: A per-request list that maps logical cache positions to physical blocks in GPU memory.
Continuous batching: Scheduling at every model step, so finished requests leave and waiting ones join without waiting for the whole batch.
Token budget: The most tokens the scheduler may process in one step, set by `--max-num-batched-tokens`.
Chunked prefill: Splitting a long prompt across several steps so it shares each step with running decodes.
Prefix caching: Reusing cached blocks when a new request starts with exactly the same tokens as an earlier one.
Preemption: Pausing a running request and freeing its cache blocks when memory runs out, then recomputing it later.
Speculative decoding: Proposing several tokens cheaply and checking them all in one pass of the full model.
TTFT and TPOT: Time to first token and time per output token: the two latencies users feel most.
Goodput: Requests per second that meet every latency target, the measure used to choose a configuration.

Syllabus

Optional field project

Launch and tune a support-assistant API on vLLM

01Stand up a baseline you can reproduce
02Measure memory, scheduling, and reuse
03Add structured output, tools, adapters, and a cheaper precision
04Find capacity and prepare for production

View optional project brief