Skip to contentModel development libraries
- PyTorch 2.14 documentationTensor operations, autograd, modules, training, compilation, distribution, and export.DocsPyTorch
- Transformers 5.17 documentationModels, tokenizers, processors, generation, Trainer, and optimization interfaces.DocsHugging Face
- PyTorch tutorialsWorked introductions and advanced investigations for the optional local exercises.TutorialsPyTorch
- Inference EngineeringPhilip Kiely’s full-stack guide to the model, hardware, runtime, optimization, and operational decisions behind production inference.BookBaseten Books
- LLM Inference HandbookA practical guide to inference fundamentals, performance metrics, optimization, GPU architecture, and production deployment, with interactive calculators and simulators.HandbookModular
- Attention Is All You NeedThe original Transformer paper. Read it for the architecture that modern language-model serving systems spend most of their effort executing efficiently.PaperarXiv 1706.03762
- LLM Inference Serving: Survey of Recent Advances and OpportunitiesA map of the serving-system landscape, including batching, memory management, parallelism, scheduling, and distributed deployment.SurveyarXiv 2407.12391
- Efficiently Scaling Transformer InferenceConnects latency, model utilization, quantization, context length, and multidimensional partitioning through an analytical performance model.PaperarXiv 2211.05102
- Orca: A Distributed Serving System for Transformer-Based Generative ModelsIntroduces iteration-level scheduling and selective batching, two foundations for understanding continuous batching in modern inference engines.PaperOSDI 2022
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessExplains why memory traffic matters as much as arithmetic and how tiling reduces transfers between high-bandwidth and on-chip memory.PaperarXiv 2205.14135
- Efficient Memory Management for Large Language Model Serving with PagedAttentionThe vLLM paper on paged KV-cache allocation, fragmentation reduction, cache sharing, and the throughput unlocked by better memory management.PaperSOSP 2023
- SGLang: Efficient Execution of Structured Language Model ProgramsIntroduces SGLang’s frontend/runtime split, RadixAttention prefix reuse, and compressed finite-state machines for structured generation.PaperarXiv 2312.07104
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-ServeShows how chunked prefill and stall-free schedules limit prefill interference while preserving decode throughput and tail latency.PaperOSDI 2024
- DistServe: Disaggregating Prefill and DecodingSeparates prefill and decode onto different GPUs, then co-optimizes placement and parallelism against TTFT and TPOT objectives.PaperOSDI 2024
Implementation references
- vLLM v0.28.0 documentationVersioned API, engine, serving, model, quantization, parallelism, observability, and benchmarking documentation.DocsvLLM
- SGLang documentationRuntime, server, API, RadixAttention, structured generation, model support, benchmarking, and deployment references.DocsSGLang
- SGLang v0.5.18 releaseThe immutable release companion for the academy’s SGLang course; pair it with the continuously updated official documentation.ReleaseGitHub
- NVIDIA Dynamo v1.4.2 documentationVersioned guidance for distributed inference, KV-aware routing, disaggregation, backend integration, deployment, and operations.DocsNVIDIA
- TensorRT-LLM documentationA complementary reference for high-performance NVIDIA inference APIs, quantization, KV-cache features, parallelism, and serving.DocsNVIDIA