Module 03
The KV Cache
Design and measure the single most important inference optimisation. From why caching exists through the arithmetic, the memory wall, attention variants built to shrink it, fragmentation and paged allocation, prefix caching and radix trees, to hit rate as the production metric that drives both cost and latency.
11 lessons · 10 videos · 1h 25m
- 03.01
Why Cache Anything
Measure the speedup from KV caching directly. Benchmark an uncached generation against a cached one on GPT-2, observe that speedup is a function of output length, and explain why a short generation shows almost nothing.
- 03.02
What Is Actually Stored
Explain that the KV cache stores past keys and values --- not queries --- and why. Trace the projections through a single attention head and identify which tensors are saved and which are recomputed.
- 03.03
Cache Arithmetic
Compute KV cache size from the formula: 2 × layers × heads × head_dim × seq_len × element_bytes. Calculate the cache for a 7B model at 2K and 32K context, compare with the device's memory, and explain why cache dominates above 32K tokens.
- 03.04
The Memory Wall
Explain why KV cache memory dominates inference cost above ~32K context, reframing long-context serving as a memory product rather than a model capability. Understand why context length is priced the way it is.
- 03.05
MHA, MQA, GQA
Compare multi-head, multi-query, and grouped-query attention. Explain that these are memory decisions --- not architecture choices --- driven by the need to shrink the KV cache. Understand how GQA shares key-value heads across query heads.
- 03.06
Fragmentation
Explain the KV cache fragmentation problem: a finished request leaves variable-sized holes that block larger allocations even though total free memory is sufficient. Connect it to classical OS memory fragmentation.
- 03.07
Paged Attention
Design paged attention: map logical KV blocks to physical ones via a block table, eliminate fragmentation, and explain why this is the 2026 floor, not an optimisation. Trace vLLM's block-level memory management from allocation through eviction.
- 03.08
Prefix Caching
Implement prefix caching: cache processed KV blocks and reuse them when a new request shares the same prefix. Explain the design rule that follows --- stable content first, variable content last. A learner who puts the user query before the system prompt has destroyed the cache without knowing it.
- 03.09
RadixAttention
Compare SGLang's radix tree approach to vLLM's hash-based prefix caching. Trace the radix data structure through radix_cache.py, radix_attention.py, and memory_pool.py. Two designs, different bets --- understand the tradeoffs without choosing a winner.
- 03.10
Hit Rate
Measure KV cache hit rate and translate it to cost and latency. Understand why production agent engineers call this the single most important metric --- a prefix-cache-aware scheduler delivers 57× faster response and 2× throughput on identical hardware.
- 03.11
Eviction
Compare KV cache eviction policies: LRU, LFU, and hybrid strategies. Explain why LRU is the dominant default and also fragile under changing traffic. Recognise that this is live research, not settled practice.