Course 04
Inference Engineering --- how models run in production
How models actually run - the generation loop, prefill and decode, batching, KV cache, the serving stack, and the workloads that changed the job.
For engineers who have trained a model and now need it to serve users - latency, throughput, the hardware budget.
- 8
- modules
- 94
- lessons
- 93
- curated videos
- 15h 20m
- of video
The Generation Loop
Understand how an LLM produces text token by token --- from the forward pass and logits through sampling and stopping to the serving stack that puts it all in production.
The Hardware Floor
Measure and model GPU performance from first principles --- throughput vs latency, FLOPs and bytes, the memory hierarchy, arithmetic intensity, the roofline model, and why decode is memory bound. By the end you can read a profiler trace and name exactly which regime your bottleneck lands in.
The KV Cache
Design and measure the single most important inference optimisation. From why caching exists through the arithmetic, the memory wall, attention variants built to shrink it, fragmentation and paged allocation, prefix caching and radix trees, to hit rate as the production metric that drives both cost and latency.
The Batch
Turn a model server into a scheduler. One request at a time wastes the machine; this module covers continuous batching, chunked prefill and prefill/decode disaggregation, then the latency numbers that say whether any of it worked.
Fewer Bytes, Fewer Steps
There are only two levers on decode: move fewer bytes per step, or take fewer steps per token. Quantisation, FlashAttention and speculative decoding are all one or the other, and each has a regime where it makes things worse.
The Fleet
One GPU becomes many, and many users become one bill. Sharding a model across devices, serving mixture-of-experts, routing requests so the cache still hits, and arriving at a defensible cost per million tokens.
Serving Agents
An agent is not a chatbot. It is hundreds of turns against one enormous shared prefix, with a tool call stalling the loop in the middle of every one. That workload breaks the batching and cache assumptions modules 3 and 4 were built on, and turns the KV cache from a buffer into a storage tier.
Thinking Costs Tokens
A model that thinks before it answers moves the compute from training to serving, makes output length unpredictable, and drags reproducibility and the reinforcement-learning loop in behind it. Every number in modules 2 and 4 has to be recomputed for a request that emits thirty thousand tokens nobody reads.
Curated from 15 channels
Every video is a public YouTube video. We pick the single clearest explanation for each lesson and credit the channel that made it.
- Stanford Online20
- GPU MODE17
- PyTorch16
- Neural Magic8
- Anyscale6
- NVIDIA Developer6
- Andrej Karpathy4
- USENIX4
- 3Blue1Brown3
- Hugging Face3
- Modal2
- AI Engineer1