Inference EngineeringAcademy
HomeFoundations Free
Courses
Pricing
Resources
JobsCoding practiceStart learning

NVIDIA Dynamo

Run a fleet of inference engines as one service with NVIDIA Dynamo. Put one frontend in front of many vLLM or SGLang workers, route each request to the worker that already holds its KV cache, split prefill and decode across GPUs, deploy and scale on Kubernetes, and keep serving when workers fail.

  • Intermediate → advanced
  • 12 units · approximately 14 hours

How to work through this course

Before you begin: The free Foundations course and one engine course (vLLM or SGLang), or equivalent experience running a single inference server. You should be comfortable with Python, a Linux shell, and basic Kubernetes. The deployment runbooks need NVIDIA GPUs, two for the multi-worker steps; every lesson exercise runs on a laptop with Python.

By the end: Size a multi-worker deployment from workload facts, choose between aggregated, KV-routed, and disaggregated serving with benchmarks you can defend, and operate it with metrics, traces, load shedding, draining, and a rehearsed rollback.

  1. Understand the idea. Read each section in order and study its diagrams. When a pause-and-predict question appears, commit to an answer before you reveal it. Look up unfamiliar terms in the glossary below.
  2. Try it yourself. Run the code examples and compare your output with the lesson's. Use the browser labs where available; steps that need a GPU are optional and say so.
  3. Check your understanding. Answer the knowledge check at the end of each lesson and make sure you can say why the other options are wrong. Complete all units to unlock the final quiz; the passing score is 80%.

The optional project brings the lessons together. Keep notes on your choices, results, and any failures so you can explain what you learned.

Course orientation

What is NVIDIA Dynamo?

NVIDIA Dynamo is an open-source framework for serving a model across many GPUs and nodes. It does not run the model itself: vLLM, SGLang, or TensorRT-LLM still execute it inside each worker. Dynamo adds the fleet layer around them: one OpenAI-compatible frontend, a registry of live workers, a router that picks a worker for each request, KV cache transfer between workers and memory tiers, and a planner and Kubernetes operator that size the worker pools.

The problem it solves

One engine schedules requests inside one replica. Once you run several replicas, questions appear that no engine answers: which worker should take this request, where its reusable KV cache already lives, whether long prompts should run on different GPUs from token generation, how many workers of each kind to run, and what happens to requests in flight when a worker dies. Dynamo answers each question with a separate component, so you adopt only the ones your workload needs.

Request path

  1. 01Frontend and router

    The frontend accepts OpenAI-compatible requests on port 8000, applies the chat template, tokenizes the prompt, and passes token IDs to its router, which picks a worker by round-robin or by cached prefix and load.

  2. 02Discovery and transports

    Workers register their endpoints in discovery (etcd, local files, or Kubernetes resources). Requests travel to workers over TCP by default, and KV cache events travel on a separate event plane.

  3. 03Engine workers

    Each worker wraps an unchanged vLLM, SGLang, or TensorRT-LLM engine at the version the Dynamo release pins. A worker either runs prefill and decode together (aggregated) or specializes in one of them (disaggregated).

  4. 04KV transfer and tiers

    NIXL moves KV blocks directly from a prefill GPU to a decode GPU, and KVBM offloads blocks to CPU memory or disk so a later request can reuse them instead of recomputing.

  5. 05Control and operations

    The Kubernetes operator deploys the whole graph, the planner recommends or applies worker counts, and metrics, traces, load shedding, draining, and request migration keep the service observable and recoverable.

What it covers

  • One client-facing API for many workers, including templating, tokenization, streaming, and cancellation.
  • Worker registration, discovery, and readiness, so requests go only to workers that can serve them.
  • KV-aware routing that weighs each worker's cached prefix against its load, using events the workers publish.
  • Disaggregated prefill and decode, with KV transfer through NIXL and cache offload through KVBM.
  • Kubernetes deployment, SLA-driven planning, load shedding, graceful shutdown, and request migration.

Outside its scope

  • Running the model. Kernels, batching inside a worker, and the local KV cache belong to the engine, at the engine version Dynamo 1.4.2 pins (vLLM 0.26.0, SGLang 0.5.16).
  • Deciding that a component is worth it. Every component adds processes and failure modes, and a single engine or plain replicas are often the right answer.
  • Authenticating users. The frontend has no user authentication of its own, so it belongs behind a gateway that has.
  • Sizing a fleet from simulation alone. DynoSim and Mocker help rank designs; only measurements on real GPUs size them.
Key terms in this course
Frontend
The OpenAI-compatible HTTP entry point that templates, tokenizes, routes, and streams responses.
Worker
A process that wraps one engine and registers its endpoints so the frontend can find it.
Discovery
The live registry of workers and their endpoints, kept in etcd, local files, or Kubernetes resources.
Aggregated serving
Each worker runs both prefill and decode for its requests, as a single engine does.
KV-aware routing
Choosing a worker by how much of the prompt it already has cached, weighed against how busy it is.
KV events
Messages workers publish when they store or evict KV blocks, which keep the router's view of each cache current.
Disaggregated serving
Running prefill and decode on separate workers and moving each request's KV cache between them.
NIXL
NVIDIA's transfer library, which moves KV blocks between GPUs and nodes.
KVBM
Dynamo's KV block manager, which offloads cache blocks from GPU memory to CPU memory and disk.
Planner
A control loop that turns measured load and latency targets into worker counts, run in advisory mode first.
Request migration
Continuing an in-flight request on another worker after its worker fails, starting from the tokens generated so far.
TTFT and ITL
Time to first token and inter-token latency: how long a user waits for the first token, and between later ones.

Syllabus

  1. 02The Dynamo frontend: APIs, streaming, and request hints75 minProOpen
  2. 03Discovery, transports, and worker readiness60 minProOpen
  3. 04Backend integrations: vLLM, SGLang, and TensorRT-LLM inside Dynamo65 minProOpen
  4. 05KV-aware routing: prefix overlap against load65 minLabProOpen
  5. 06NIXL, KVBM, and where KV blocks live70 minProOpen
  6. 07Disaggregated prefill and decode75 minProOpen
  7. 08Deploy locally with an engine backend60 minProOpen
  8. 09Operate Dynamo on Kubernetes65 minProOpen
  9. 10Plan capacity and autoscale with the Planner75 minProOpen
  10. 11Observability and resilience: trace, shed, drain, and recover80 minProOpen
  11. 12Capstone: design the inference fleet75 minProOpen
  12. QFinal graded quiz20 questions · Pass at 80%ProComplete all units to unlock

Optional field project

Build, measure, and defend a Dynamo fleet

  1. 01Run one frontend and one worker, then prove membership and the API
  2. 02Measure routing, cache tiers, and the prefill and decode split
  3. 03Make the topology reproducible, then run it on Kubernetes
  4. 04Plan capacity, drill failures, and defend the design
View optional project brief

Sign in to read the project brief.