03.04 · Concept
The Memory Wall
Explain why KV cache memory dominates inference cost above ~32K context, reframing long-context serving as a memory product rather than a model capability. Understand why context length is priced the way it is.
KV cache memory becomes the limiting resource for long-context inference because every active token leaves keys and values that must stay resident and be read during decoding. Beyond roughly 32K context, providers are mainly allocating high-bandwidth memory capacity, bandwidth, cache placement, and residency time, not just exposing a model feature.
What this lesson answers
- why does long context cost so much
- what makes KV cache dominate inference cost
- why is long context a memory problem
Notes
The KV cache is the persistent tensor of past attention keys and values that converts autoregressive decoding from recomputing all prior projections to appending one new row per layer per generated token. For a decoder-only transformer with layers, KV heads, head dimension , bytes per element , batch size , and resident context length , its size is , where the factor is for keys and values. This formula, not the parameter count alone, is the governing equation for long-context serving.
Common questions
- Why does KV cache matter more at long context lengths?
- During decoding, the model generates one new token but still attends over the stored keys and values from the existing context. As the context grows, that resident cache grows with it, and the read traffic grows too. The bottleneck shifts from arithmetic to keeping enough high-bandwidth memory available and streaming it fast enough.
- Does FlashAttention solve the long-context memory wall?
- No. FlashAttention improves attention execution by reducing temporary memory use and making better use of on-chip memory, especially during prefill. It does not remove the need to store past keys and values for decoding. At long context lengths, the KV cache remains the working set that must be stored, read, scheduled, and sometimes shared.
- Why do providers price long context by input length?
- Longer inputs reserve more KV cache for longer periods and require more memory bandwidth during generation. That resource is scarce GPU memory, not just extra compute. Pricing by input length reflects the cost of admitting the request, keeping its cache resident, and reading that cache repeatedly while tokens are produced.
Short definition: what is Memory Wall?