SGLang

Run SGLang in production. Serve models over HTTP and Python, reuse shared prompt prefixes with the radix cache, constrain output and parse tool calls, scale across GPUs, and benchmark every change with evidence you can defend.

Beginner → production
13 units · approximately 14 hours

How to work through this course

Before you begin: The free Foundations course or equivalent knowledge: tokens, prefill and decode, the KV cache, and serving metrics. You should be comfortable with Python and a Linux shell. The deployment runbooks need an NVIDIA GPU; the lesson exercises run on a laptop with Python and NumPy.

By the end: Launch and operate an SGLang service, predict when prefix reuse will cut prefill work, validate structured output and tool calls in your own code, and decide on speculative, quantized, or parallel setups with controlled benchmarks.

Understand the idea. Read each section in order and study its diagrams. When a pause-and-predict question appears, commit to an answer before you reveal it. Look up unfamiliar terms in the glossary below.
Try it yourself. Run the code examples and compare your output with the lesson's. Use the browser labs where available; steps that need a GPU are optional and say so.
Check your understanding. Answer the knowledge check at the end of each lesson and make sure you can say why the other options are wrong. Complete all units to unlock the final quiz; the passing score is 80%.

The optional project brings the lessons together. Keep notes on your choices, results, and any failures so you can explain what you learned.

Course orientation

What is SGLang?

SGLang is an open-source serving engine for large language models and multimodal models. Its runtime, SRT (SGLang Runtime), accepts requests over HTTP or from Python, batches them continuously on one or more GPUs, keeps their KV cache in a radix tree so that shared prompt prefixes are computed once, and streams tokens back. SGLang also includes a frontend language for programs that make several model calls.

The problem it solves

Real traffic repeats itself. Chat turns resend the whole conversation, agents resend the same tool instructions, and few-shot prompts share their examples. A server that recomputes that shared text for every request spends GPU time on prefill and makes users wait longer for the first token. SGLang finds identical token prefixes and reuses their KV cache while it batches many requests, and it constrains and parses outputs for structured and agent workloads.

Request path

01HTTP server and tokenizer manager
The API server accepts OpenAI-compatible and native requests, applies the chat template, tokenizes the prompt, and passes the request to the scheduler.
02Scheduler and radix cache
The scheduler matches each prompt against the radix tree of cached prefixes, admits requests when KV cache memory allows, and forms every prefill and decode batch.
03Model workers
Workers run the forward pass on one or more GPUs, using the chosen attention backend, CUDA graphs, and tensor, pipeline, data, or expert parallelism. Grammar masks restrict sampling when output must match a schema.
04Detokenizer and response
The detokenizer turns token IDs back into text and streams it to the client. Parsers separate reasoning from the answer and extract tool calls in the formats a model family uses.

What it covers

Scheduling requests into continuous prefill and decode batches and admitting them against KV cache memory.
Reusing the KV cache of identical token prefixes across requests, and evicting the least recently used entries when memory runs short.
Constrained decoding for JSON schemas, regular expressions, and grammars, plus parsers for reasoning and tool-call formats.
Running models across GPUs with tensor, pipeline, data, and expert parallelism, and speeding up decode with CUDA graphs, quantization, and speculative decoding.

Outside its scope

Matching prompts by meaning. The radix cache reuses only identical token prefixes, so a reworded or reordered prompt misses.
Deciding whether a tool call is allowed. SGLang returns a proposed call, and your application must validate and authorize it before anything runs.
Training or fine-tuning models. SGLang serves weights and LoRA adapters that were trained elsewhere.
Promising that every combination of features works together. Test each combination you plan to use on the pinned release.

Key terms in this course

SRT: SGLang Runtime: the serving engine that schedules, batches, and executes requests.
RadixAttention: SGLang's way of reusing KV cache for shared prompt prefixes, organized as a radix tree.
Radix tree: A tree of token sequences in which requests that start with the same tokens share one path, and therefore one copy of that KV cache.
Prefix hit: The leading part of a prompt whose KV cache is already in the radix tree, so prefill can skip it.
Chunked prefill: Splitting a long prompt's prefill into pieces so that other requests' decode steps are not stuck behind it.
Constrained decoding: Masking tokens during sampling so the output must match a JSON schema, a regular expression, or a grammar.
Speculative decoding: Drafting several tokens cheaply and checking them in one forward pass of the main model, which can cut the time per output token.
TTFT and ITL: Time to first token and inter-token latency: how long a user waits for the first token, and between later ones.
Goodput: The number of requests per second that succeed and meet every latency target.
PD disaggregation: Running prefill and decode on separate workers and moving each request's KV cache between them.

Syllabus

Optional field project

Launch and tune an SGLang support assistant

01Launch the service and trace one request
02Measure prefix reuse and scheduling
03Validate structured output and optional features
04Benchmark, scale, and operate the service

View optional project brief