Inference Engineering --- how models run in production - full curriculum

Understand how an LLM produces text token by token --- from the forward pass and logits through sampling and stopping to the serving stack that puts it all in production.

Measure and model GPU performance from first principles --- throughput vs latency, FLOPs and bytes, the memory hierarchy, arithmetic intensity, the roofline model, and why decode is memory bound. By the end you can read a profiler trace and name exactly which regime your bottleneck lands in.

Design and measure the single most important inference optimisation. From why caching exists through the arithmetic, the memory wall, attention variants built to shrink it, fragmentation and paged allocation, prefix caching and radix trees, to hit rate as the production metric that drives both cost and latency.

Turn a model server into a scheduler. One request at a time wastes the machine; this module covers continuous batching, chunked prefill and prefill/decode disaggregation, then the latency numbers that say whether any of it worked.

There are only two levers on decode: move fewer bytes per step, or take fewer steps per token. Quantisation, FlashAttention and speculative decoding are all one or the other, and each has a regime where it makes things worse.

One GPU becomes many, and many users become one bill. Sharding a model across devices, serving mixture-of-experts, routing requests so the cache still hits, and arriving at a defensible cost per million tokens.

An agent is not a chatbot. It is hundreds of turns against one enormous shared prefix, with a tool call stalling the loop in the middle of every one. That workload breaks the batching and cache assumptions modules 3 and 4 were built on, and turns the KV cache from a buffer into a storage tier.

A model that thinks before it answers moves the compute from training to serving, makes output length unpredictable, and drags reproducibility and the reinforcement-learning loop in behind it. Every number in modules 2 and 4 has to be recomputed for a request that emits thirty thousand tokens nobody reads.