Inference EngineeringAcademy
HomeFoundations Free
Courses
Pricing
Resources
JobsCoding practiceStart learning

Inference Engineering Foundations

Learn how a language model turns a prompt into a streamed answer, and what each step costs in memory and time. Work through real numbers for Llama 3.1 8B, runnable NumPy code, and the measurements serving teams rely on.

  • Beginner → advanced
  • 9 units · self-paced

How to work through this course

Before you begin: Comfortable with everyday Python. The lessons introduce the math and the NumPy you need as you go.

By the end: Follow a request through every stage, calculate its memory and compute needs, and find which part of the system is causing a delay.

  1. Understand the idea. Read each section in order and study its diagrams. When a pause-and-predict question appears, commit to an answer before you reveal it. Look up unfamiliar terms in the glossary below.
  2. Try it yourself. Run the code examples and compare your output with the lesson's. Use the browser labs where available; steps that need a GPU are optional and say so.
  3. Check your understanding. Answer the knowledge check at the end of each lesson and make sure you can say why the other options are wrong. Complete all units to unlock the final quiz; the passing score is 80%.

The optional project brings the lessons together. Keep notes on your choices, results, and any failures so you can explain what you learned.

Course orientation

What is inference engineering?

Inference engineering is the work of running trained models quickly, cheaply, and reliably for real users. It covers everything between a user pressing Enter and the last token arriving: turning text into tokens, scheduling work on the GPU, holding each request's state in memory, streaming results back, and recovering when something fails.

The problem it solves

Getting one answer from a model in a notebook is easy. Serving thousands of people at once is not: prompts vary from a few words to whole books, GPU memory runs out, users expect the first word within a second, and every idle GPU second costs money. This course connects the model's arithmetic to those practical limits, so you can predict and fix them.

Request path

  1. 01Request boundary

    A chat template and a tokenizer turn the user's messages into token IDs, and the request's settings (length limits, sampling, stop rules) come along with them.

  2. 02Model execution

    Prefill runs the whole prompt through the model in one parallel pass. Decode then produces one new token per step until the answer is complete.

  3. 03Memory and scheduling

    A scheduler shares the GPU between many requests, and the KV cache keeps each request's attention state so earlier tokens are never recomputed.

  4. 04Service behavior

    Sampling, stop rules, streaming, cancellation, metrics, and error handling turn a stream of token IDs into an API people can depend on.

What it covers

  • Measuring and improving latency, throughput, memory use, and cost per token.
  • Deciding how requests are batched, scheduled, admitted, and given cache memory.
  • Streaming, cancellation, monitoring, and cleaning up after every request.

Outside its scope

  • Training or fine-tuning model weights. The PyTorch and Transformers courses cover that.
  • Prompt writing. Better prompts do not fix memory limits, queues, or slow decode steps.
  • A bare HTTP wrapper around a model. A real service still needs admission control, scheduling, and cleanup.
Key terms in this course
Token
The unit a model reads and writes: a word, part of a word, or a symbol, stored as an integer ID.
Tensor
An array of numbers with a shape, a data type, and a device. Weights, activations, and the KV cache are all tensors.
Prefill
The phase that processes all prompt tokens in one parallel pass and fills the KV cache.
Decode
The phase that generates one new token per step, reusing the KV cache.
KV cache
The stored attention keys and values of every earlier token, kept so decode does not recompute them.
Batch
A group of requests the GPU processes together, so one read of the weights serves all of them.
Quantization
Storing weights or the cache with fewer bits per number, such as 8 or 4 instead of 16, to save memory and time.
TTFT
Time to first token: from the moment a request arrives until its first token reaches the user.
ITL
Inter-token latency: the time between two consecutive streamed tokens.
Throughput
How much useful work the system finishes per second, such as output tokens or completed requests.

Syllabus

  1. 02Tensors, shapes, and the forward pass50 minLabOpen
  2. 03From text to model inputs50 minOpen
  3. 04Attention and the transformer block60 minLabOpen
  4. 05Generation is a loop, not one pass50 minOpen
  5. 06The KV cache and its memory cost55 minLabOpen
  6. 07Batching and scheduling: one weight read, many requests55 minOpen
  7. 08Quantization: fewer bits, measured error55 minOpen
  8. 09Serve, measure, and optimize the bottleneck80 minOpen
  9. QFinal graded quiz20 questions · Pass at 80%Complete all units to unlock

Optional field project

Plan and measure a chat service

  1. 01Trace one request through the model
  2. 02Explain tokens, attention, and sampling
  3. 03Size memory for the real workload
  4. 04Measure the service and recommend one change
View optional project brief

Sign in to read the project brief.