Curation in progress
Inference Engineering
Eight modules on the half of machine learning that runs in production. How a token is actually produced, why the GPU spends most of its time waiting, what the KV cache costs, how batching and scheduling fill the machine, how quantisation and speculative decoding shrink the work, and what a fleet of GPUs costs per million tokens. Then the two workloads that changed the job: serving agents, where hundreds of turns share one enormous prefix, and serving models that think before they answer.
- Modules
- 8
- Lessons
- 94
- Watch time
- 15h 20m
- Free to open
- 16
The Generation Loop
Understand how an LLM produces text token by token --- from the forward pass and logits through sampling and stopping to the serving stack that puts it all in production.
- 01.01Forward Pass vs GenerationDistinguish training forward passes from autoregressive generation and explain why inference is a fundamentally different workload.
- 01.02The Generation LoopTrace the autoregressive generation loop: forward pass → sample token → append → repeat. Read Karpathy's generate function in nanoGPT.
- 01.03Prefill & DecodeExplain the two distinct phases of LLM inference: compute-bound prefill and memory-bound decode. Understand why they must be treated differently.
- 01.04LogitsRead the raw logit vector from a forward pass, understand how it maps vocabulary to scores, and trace the unembedding step.
- 01.05SamplingImplement temperature scaling, top-k, top-p, and greedy sampling. Explain what each hyperparameter controls and when to use it.
- 01.06StoppingImplement stopping criteria: EOS token, max tokens, stop sequences. Understand why stopping is a correctness concern, not a convenience.
- 01.07Tokenisation at Serving TimeDiagnose tokenisation issues in production: token count vs character count, non-English compression, and special token handling at the serving boundary.
- 01.08The Context BudgetCalculate KV cache memory from model parameters and context length. Reason about the quadratic prefill cost and why longer contexts change the inference profile.
- 01.09First BatchUnderstand batching as a throughput lever: how multiple sequences share a forward pass, why batch size is a latency/throughput tradeoff, and how a scheduler picks requests to batch together.
- 01.10The Serving StackMap the components of a production serving system: model runner, scheduler, KV cache manager, request queue, and REST/gRPC frontend.
- 01.11Your BaselineRun a benchmark on your own hardware: throughput, time-to-first-token, inter-token latency. Establish a reproducible measurement before any optimisation.
The Hardware Floor
Measure and model GPU performance from first principles --- throughput vs latency, FLOPs and bytes, the memory hierarchy, arithmetic intensity, the roofline model, and why decode is memory bound. By the end you can read a profiler trace and name exactly which regime your bottleneck lands in.
- 02.01Why the GPU WaitsUnderstand the GPU as a throughput machine working against a latency-optimized CPU. Explain the three performance regimes: compute bound, memory bandwidth bound, and overhead bound.
- 02.02FLOPs and BytesCalculate the FLOP count and memory traffic for a single transformer forward pass. Understand why resource accounting is the first step in any performance investigation.
- 02.03The Memory HierarchyTrace data through the GPU memory hierarchy: DRAM, L2 cache, shared memory, registers. Explain why each level exists and what it costs to cross.
- 02.04Arithmetic IntensityCompute arithmetic intensity --- operations per byte --- for a kernel. Use it to predict whether a workload is compute bound or memory bound before running it.
- 02.05The RooflineDraw and read a roofline plot. Place a kernel on it, identify the ridge point, and determine which side of the ridge it sits on.
- 02.06Why Decode Is Memory BoundProve from FLOP and byte counts that a single decode step reads far more bytes than it computes operations. Explain why larger batches are the only lever that helps.
- 02.07Where the Memory WentBuild a memory budget for a transformer inference run: weights, KV cache, activations. Identify which component dominates at which batch size and context length.
- 02.08PrecisionCompare FP32, FP16, BF16, INT8, and FP4. Understand the memory bandwidth savings and the accuracy tradeoffs. Read a dtype annotation on a model checkpoint and know what it implies for throughput.
- 02.09Reading the DeviceInterpret nvidia-smi output. Understand what GPU-Util actually measures --- and what it does not. Diagnose why a GPU reporting 100% utilisation can still be memory bound.
- 02.10Profiling a Forward PassUse the PyTorch profiler to capture a trace of a forward pass. Read the trace: identify CPU-GPU synchronisation gaps, kernel durations, and memory transfers.
- 02.11Launch OverheadMeasure kernel launch overhead. Apply the batch doubling test: double the work and if runtime barely rises, the kernel is overhead bound. Recognise when CUDA graphs eliminate the problem.
- 02.12Naming Your BottleneckSynthesise: given a profiler trace, compute arithmetic intensity, place the kernel on a roofline, and name whether it is compute, bandwidth, or overhead bound. Defend the answer with numbers.
The KV Cache
Design and measure the single most important inference optimisation. From why caching exists through the arithmetic, the memory wall, attention variants built to shrink it, fragmentation and paged allocation, prefix caching and radix trees, to hit rate as the production metric that drives both cost and latency.
- 03.01Why Cache AnythingMeasure the speedup from KV caching directly. Benchmark an uncached generation against a cached one on GPT-2, observe that speedup is a function of output length, and explain why a short generation shows almost nothing.
- 03.02What Is Actually StoredExplain that the KV cache stores past keys and values --- not queries --- and why. Trace the projections through a single attention head and identify which tensors are saved and which are recomputed.
- 03.03Cache ArithmeticCompute KV cache size from the formula: 2 × layers × heads × head_dim × seq_len × element_bytes. Calculate the cache for a 7B model at 2K and 32K context, compare with the device's memory, and explain why cache dominates above 32K tokens.
- 03.04The Memory WallExplain why KV cache memory dominates inference cost above ~32K context, reframing long-context serving as a memory product rather than a model capability. Understand why context length is priced the way it is.
- 03.05MHA, MQA, GQACompare multi-head, multi-query, and grouped-query attention. Explain that these are memory decisions --- not architecture choices --- driven by the need to shrink the KV cache. Understand how GQA shares key-value heads across query heads.
- 03.06FragmentationExplain the KV cache fragmentation problem: a finished request leaves variable-sized holes that block larger allocations even though total free memory is sufficient. Connect it to classical OS memory fragmentation.
- 03.07Paged AttentionDesign paged attention: map logical KV blocks to physical ones via a block table, eliminate fragmentation, and explain why this is the 2026 floor, not an optimisation. Trace vLLM's block-level memory management from allocation through eviction.
- 03.08Prefix CachingImplement prefix caching: cache processed KV blocks and reuse them when a new request shares the same prefix. Explain the design rule that follows --- stable content first, variable content last. A learner who puts the user query before the system prompt has destroyed the cache without knowing it.
- 03.09RadixAttentionCompare SGLang's radix tree approach to vLLM's hash-based prefix caching. Trace the radix data structure through radix_cache.py, radix_attention.py, and memory_pool.py. Two designs, different bets --- understand the tradeoffs without choosing a winner.
- 03.10Hit RateMeasure KV cache hit rate and translate it to cost and latency. Understand why production agent engineers call this the single most important metric --- a prefix-cache-aware scheduler delivers 57× faster response and 2× throughput on identical hardware.
- 03.11EvictionCompare KV cache eviction policies: LRU, LFU, and hybrid strategies. Explain why LRU is the dominant default and also fragile under changing traffic. Recognise that this is live research, not settled practice.
The Batch
Turn a model server into a scheduler. One request at a time wastes the machine; this module covers continuous batching, chunked prefill and prefill/decode disaggregation, then the latency numbers that say whether any of it worked.
- 04.01One Request at a TimeExplain why batch size 1 is the worst case for a GPU: the whole weight matrix is read from HBM to produce a single token, so arithmetic intensity collapses and the device runs at a few percent of peak.
- 04.02Static BatchingDescribe request-level batching and name its two costs: padding to the longest sequence, and head-of-line blocking where every finished request waits for the slowest one.
- 04.03Continuous BatchingExplain iteration-level scheduling: admitting arriving requests and retiring finished ones at every decode step rather than every request, and why that alone is the largest single throughput win in LLM serving.
- 04.04The Scheduler LoopTrace what a serving engine does on one step: the waiting and running queues, admission against the KV budget, preemption and recomputation when memory runs short.
- 04.05Chunked PrefillExplain how splitting a long prompt into token-budgeted chunks lets prefill and decode share a batch, so a single long prompt stops stalling every other user's stream.
- 04.06Prefill/Decode InterferenceShow why one GPU doing both phases does neither well: prefill saturates compute, decode saturates bandwidth, and mixing them means each phase pays the other's stall.
- 04.07DisaggregationDescribe running prefill and decode on separate GPU pools so each scales independently, and state honestly when it does not pay: short prompts, low concurrency, or fewer GPUs than it takes to keep both pools busy.
- 04.08Moving the KV CacheExplain why the KV transfer between a prefill worker and a decode worker is the engineering crux of disaggregation, and how NVLink, RDMA and a transfer layer decide whether the split is a win or a regression.
- 04.09TTFT, TPOT, ITLDefine time to first token, time per output token and inter-token latency, and say which one a user actually feels in a streaming chat versus a batch job.
- 04.10Throughput vs LatencyRead the batch-size curve: larger batches raise tokens per second and raise per-user latency at the same time. Pick a point on it deliberately instead of inheriting a default.
- 04.11Goodput and SLOsUse goodput instead of raw throughput: requests per second that met both the TTFT and the TPOT target. Set a latency budget a serving change can be measured against.
- 04.12Benchmarking Your ServerRun a load test that is not self-deception: a realistic prompt-length distribution, controlled concurrency, percentiles rather than means, and a warm cache reported separately from a cold one.
Fewer Bytes, Fewer Steps
There are only two levers on decode: move fewer bytes per step, or take fewer steps per token. Quantisation, FlashAttention and speculative decoding are all one or the other, and each has a regime where it makes things worse.
- 05.01Two Ways to Go FasterState the framing the rest of the module hangs on: decode is bandwidth-bound, so every real optimisation either shrinks the bytes read per step or reduces the number of steps. Classify a proposed optimisation as one or the other.
- 05.02Number FormatsCompare BF16, FP8, INT8, INT4 and the microscaling formats MXFP4 and NVFP4 by what they cost in bytes, what hardware runs them natively, and where the exponent bits went.
- 05.03Weight-Only QuantisationExplain why INT4 weight-only quantisation is the natural first move for decode, since the weights are the bytes being read, and how AWQ and GPTQ decide what to round.
- 05.04Activation QuantisationExplain why quantising activations as well as weights is harder than weights alone: a few outlier channels carry a disproportionate range, and how SmoothQuant migrates that difficulty into the weights where it is cheap.
- 05.05Calibration and QualityChoose a calibration set, pick per-channel over per-tensor scaling where it matters, and measure the accuracy cost honestly rather than quoting the paper's number.
- 05.06KV Cache QuantisationQuantise the KV cache to FP8 or NVFP4, the largest win available at long context and large batch, because the cache is what decode is actually reading. Then find the point where recall starts to degrade.
- 05.07FlashAttentionExplain FlashAttention as an IO result rather than a maths one: tiling the computation so the full attention score matrix is never written to HBM, and what FA2 and FA3 changed on Ampere and Hopper.
- 05.08Speculative DecodingExplain draft-then-verify: a cheap model proposes k tokens, the target model verifies them in one forward pass, and the rejection-sampling step makes the output distribution identical to decoding without it.
- 05.09Draft ModelsChoose a draft: a small model from the same family, an n-gram or prompt-lookup draft that costs no GPU at all, or self-speculation. Compute the acceptance rate you need for each to break even.
- 05.10EAGLE and MTPDescribe trained draft heads: Medusa's parallel heads, EAGLE's feature-level autoregression, and multi-token prediction built into the model. Explain why these, not separate draft models, are what production engines ship.
- 05.11When Speculation LosesExplain why speculative decoding is a latency technique and not a throughput one: at high batch size the GPU is already compute-saturated, so verification costs real time and a low acceptance rate makes the server slower.
- 05.12Stacking the WinsCompose quantisation, cache reuse and speculation, and measure the combination rather than adding up the individual claims: they contend for the same bandwidth and the same batch slots.
The Fleet
One GPU becomes many, and many users become one bill. Sharding a model across devices, serving mixture-of-experts, routing requests so the cache still hits, and arriving at a defensible cost per million tokens.
- 06.01When One GPU Is Not EnoughName the three separate reasons to shard: the weights do not fit, the KV cache does not fit, or the latency target is below what one device can reach. Recognise that they call for different strategies.
- 06.02Tensor ParallelismSplit a matrix multiply across devices column-wise then row-wise, account for the all-reduce every layer pays, and explain why TP stops being worth it outside one NVLink domain.
- 06.03Pipeline ParallelismSplit a model by layer across devices, describe the bubble that idle stages create, and say when PP is the right answer across nodes where TP's collectives are too expensive.
- 06.04The InterconnectCompare NVLink and NVSwitch against PCIe and InfiniBand by bandwidth and latency, and use the cost of an all-reduce as the real constraint on how far a model can be sharded.
- 06.05Mixture of ExpertsExplain why MoE changed the serving problem rather than just the training one: compute per token stays small while the memory that must be resident stays large, so the bottleneck moves.
- 06.06Expert ParallelismPlace experts across devices, trace the all-to-all that routing requires, and explain why expert load imbalance, not raw FLOPs, is what limits a wide-EP deployment.
- 06.07Routing RequestsRoute across replicas by KV-cache locality and session affinity, and explain why plain round-robin destroys the prefix hit rate the KV cache module worked to build.
- 06.08Multi-Tenancy and LoRAServe many fine-tunes from one base model by swapping low-rank adapters per request, and state what that buys over one deployment per tenant.
- 06.09Autoscaling and Cold StartsChoose between scale-to-zero and a warm pool by pricing the cold start honestly: loading tens of gigabytes of weights is the cold start, and no amount of container tuning removes it.
- 06.10Choosing the HardwareCompare H100, H200, B200 and rack-scale GB200 NVL72 on the axes that decide inference cost rather than on peak FLOPs: memory capacity, memory bandwidth, interconnect and native low-precision support.
- 06.11Cost per Million TokensBuild the unit-economics model from GPU-hour price, measured tokens per second and realistic utilisation, and produce separate input and output prices the way every provider quotes them.
- 06.12Self-Host or APIDecide between self-hosting and a hosted API on measured numbers: break-even volume, latency requirements, data residency and the engineering time a fleet actually costs. Accept that the API often wins.
Serving Agents
An agent is not a chatbot. It is hundreds of turns against one enormous shared prefix, with a tool call stalling the loop in the middle of every one. That workload breaks the batching and cache assumptions modules 3 and 4 were built on, and turns the KV cache from a buffer into a storage tier.
- 07.01The Agent WorkloadDescribe the shape of agent traffic: long multi-turn trajectories, input sequences an order of magnitude larger than the output, near-total prefix overlap between turns. Name which specific assumptions from modules 3 and 4 stop holding.
- 07.02Session AffinityTreat a session rather than a request as the unit of scheduling: measure prefix-cache hit rate across the turns of one real trajectory, and show what it costs when the second turn lands on a replica that has never seen the first.
- 07.03Cache-Aware RoutingRoute on cache residency instead of load: send a request to the replica that already holds its prefix, and explain why round-robin, the correct answer for a stateless service, is the worst possible policy here.
- 07.04The KV Memory HierarchyPlace the KV cache on a storage hierarchy: HBM, host DRAM, local NVMe, a remote pooled store. Explain why production fleets now hold more cache off the GPU than on it. This is module 2's memory hierarchy, one level up.
- 07.05KV Offload and ReloadDo the arithmetic that decides between reloading a cache and recomputing it: bytes to move against the link bandwidth, versus the prefill FLOPs to rebuild the same tokens. Find the prefix length where the answer flips.
- 07.06Tool Calls in the LoopAccount for the pause: while a tool runs, the sequence is neither prefilling nor decoding, and its KV cache is occupying memory it is not using. Compare holding, offloading and evicting it, and say which the tool latency distribution argues for.
- 07.07The Tool Schema TaxMeasure what a tool definition costs before any tool is called: a handful of schemas adds hundreds of tokens to the prompt of every single request, which is real time-to-first-token unless that block is prefix-cached. Then show the fix: put the schemas where the cache can hold them.
- 07.08Structured OutputExplain constrained decoding as a mask over the logit vector from module 1: at each step a grammar decides which tokens are legal, and sampling happens only over those. Distinguish it from asking a model nicely for JSON, which is not a guarantee.
- 07.09What Grammars CostSeparate the two costs of constrained decoding and find which one is actually hurting: applying a mask per step is cheap and constant, while COMPILING a schema into an automaton is neither, so a fleet with many distinct one-shot schemas pays a tail latency a fleet reusing a few cached ones never sees.
- 07.10Long-Context PrefillTake module 4's chunked prefill to agent scale: schedule a hundred-thousand-token prompt without starving every decoding request behind it, and show why the cache hit rate matters more here than any kernel does.
- 07.11Context CompactionCut the prompt instead of speeding it up, summarising, pruning or retrieving rather than resending. Understand the trap that pays for it: rewriting history shifts every token position after the edit, so a compaction that saves tokens can destroy the prefix cache it was meant to help.
- 07.12Agents per MegawattSize an agent fleet the way the workload demands: fix the SLOs first, then ask how many concurrent agents the machine sustains, not how many tokens per second it can emit. Build the per-session cost model, and identify the two levers that actually move it.
Thinking Costs Tokens
A model that thinks before it answers moves the compute from training to serving, makes output length unpredictable, and drags reproducibility and the reinforcement-learning loop in behind it. Every number in modules 2 and 4 has to be recomputed for a request that emits thirty thousand tokens nobody reads.
- 08.01Test-Time ComputeState the trade the field made: buy accuracy with inference tokens instead of parameters, and understand why that moves the dominant cost of an AI product from a one-off training run to a bill that arrives every day.
- 08.02What a Reasoning Model EmitsSeparate reasoning tokens from answer tokens at the serving boundary: both are generated, both are decoded, both are billed, and only one is shown. Explain what that does to a token-count budget written for a chat model.
- 08.03The Decode-Heavy WorkloadRedo module 2's arithmetic for a request whose output is two orders of magnitude longer than its input, and show that a reasoning request is almost pure decode, so every memory-bandwidth conclusion applies harder, and every prefill optimisation matters less.
- 08.04Effort and BudgetsDrive the one knob a caller is given, an effort level or a thinking-token ceiling, and plot the curve it moves along: accuracy against latency and cost. Then set it per route rather than per product, because most requests do not need it at all.
- 08.05Unpredictable Output LengthExplain why a scheduler that plans capacity from an expected output length degrades when that length varies by an order of magnitude between requests, and describe what the batch does when a handful of very long thoughts hold their slots for minutes.
- 08.06Parallel SamplingSpend the budget sideways instead of deep, on best-of-n and self-consistency, and price it properly: n branches share one prompt prefix but each own their divergent KV, so the cost is sublinear in memory and linear in decode.
- 08.07Verifiers in the Serving PathPut a second model in the request path, a verifier or reward model that scores candidate answers, and reason about the two designs: scoring the final answer, or scoring each step. Account for the fact that the verifier is inference too.
- 08.08Serving Two ModelsCo-locate a small model beside a large one, whether a verifier, a draft model or a router, and divide the device between them: weights, KV budget and scheduling priority. Show what happens to the big model's throughput when the small one is not given its own priority class.
- 08.09SLOs for ThinkingRewrite module 4's latency vocabulary for a model that is silent for the first thirty seconds: time-to-first-token measures a token nobody sees, so define what to promise instead, and decide whether to stream the reasoning at all.
- 08.10DeterminismExplain why the same prompt at temperature zero can return different text on two runs: the reduction kernels are batch-size dependent, so a request's own numerics change with who else happened to be in the batch. Know what batch-invariant mode costs and when it is worth it.
- 08.11Rollouts Are InferenceRecognise the reinforcement-learning loop as an inference problem: the trainer spends most of its wall clock inside a serving engine generating rollouts. Work through weight synchronisation, pausing without dropping in-flight requests, and why bitwise consistency between the trainer and the sampler is a correctness requirement rather than a nicety.
- 08.12Cost per Solved TaskReplace module 6's cost per million tokens with the only number a reasoning product can be judged on, cost per task actually solved, and use it to settle the real question: a small model sampled many times, or one call to a large one.