Module 05
Fewer Bytes, Fewer Steps
There are only two levers on decode: move fewer bytes per step, or take fewer steps per token. Quantisation, FlashAttention and speculative decoding are all one or the other, and each has a regime where it makes things worse.
12 lessons · 12 videos · 2h 58m
- 05.01
Two Ways to Go Faster
State the framing the rest of the module hangs on: decode is bandwidth-bound, so every real optimisation either shrinks the bytes read per step or reduces the number of steps. Classify a proposed optimisation as one or the other.
- 05.02
Number Formats
Compare BF16, FP8, INT8, INT4 and the microscaling formats MXFP4 and NVFP4 by what they cost in bytes, what hardware runs them natively, and where the exponent bits went.
- 05.03
Weight-Only Quantisation
Explain why INT4 weight-only quantisation is the natural first move for decode, since the weights are the bytes being read, and how AWQ and GPTQ decide what to round.
- 05.04
Activation Quantisation
Explain why quantising activations as well as weights is harder than weights alone: a few outlier channels carry a disproportionate range, and how SmoothQuant migrates that difficulty into the weights where it is cheap.
- 05.05
Calibration and Quality
Choose a calibration set, pick per-channel over per-tensor scaling where it matters, and measure the accuracy cost honestly rather than quoting the paper's number.
- 05.06
KV Cache Quantisation
Quantise the KV cache to FP8 or NVFP4, the largest win available at long context and large batch, because the cache is what decode is actually reading. Then find the point where recall starts to degrade.
- 05.07
FlashAttention
Explain FlashAttention as an IO result rather than a maths one: tiling the computation so the full attention score matrix is never written to HBM, and what FA2 and FA3 changed on Ampere and Hopper.
- 05.08
Speculative Decoding
Explain draft-then-verify: a cheap model proposes k tokens, the target model verifies them in one forward pass, and the rejection-sampling step makes the output distribution identical to decoding without it.
- 05.09
Draft Models
Choose a draft: a small model from the same family, an n-gram or prompt-lookup draft that costs no GPU at all, or self-speculation. Compute the acceptance rate you need for each to break even.
- 05.10
EAGLE and MTP
Describe trained draft heads: Medusa's parallel heads, EAGLE's feature-level autoregression, and multi-token prediction built into the model. Explain why these, not separate draft models, are what production engines ship.
- 05.11
When Speculation Loses
Explain why speculative decoding is a latency technique and not a throughput one: at high batch size the GPU is already compute-saturated, so verification costs real time and a low acceptance rate makes the server slower.
- 05.12
Stacking the Wins
Compose quantisation, cache reuse and speculation, and measure the combination rather than adding up the individual claims: they contend for the same bandwidth and the same batch slots.