03.03 · Walkthrough
Cache Arithmetic
Compute KV cache size from the formula: 2 × layers × heads × head_dim × seq_len × element_bytes. Calculate the cache for a 7B model at 2K and 32K context, compare with the device's memory, and explain why cache dominates above 32K tokens.
KV cache memory is linear in layers, KV heads, head size, sequence length and element size. For long-context decoding, it can exceed the weight footprint per active request, so batching and placement become memory problems before compute problems, especially around 32K-token contexts.
What this lesson answers
- how do I calculate KV cache size
- why does KV cache dominate long context inference
- how does GQA change KV cache memory
Notes
The KV cache is the per-layer record of attention keys and values saved after a token has been processed, so later decode steps do not recompute projections for the whole prefix. For a dense decoder with ordinary multi-head attention, its size is , where is key plus value, is layer count, is the number of KV heads, is head dimension, is sequence length in tokens, and is bytes per stored element.
Common questions
- What is the KV cache size formula?
- For a standard decoder-only transformer, compute it as key plus value, multiplied by layer count, KV head count, head dimension, sequence length and bytes per element. In shorthand: 2 × layers × KV heads × head_dim × seq_len × element_bytes. Use KV heads, not query heads, when the model uses GQA or MQA.
- Why can a 32K context use so much GPU memory?
- Weights are loaded once, but KV cache is allocated for every resident token in every active sequence. At 32K, a single request can consume several GiB or more just for cached attention state. Add concurrent requests, runtime buffers and allocator overhead, and the scheduler becomes constrained by usable HBM.
- Do FlashAttention or paged KV cache reduce the formula?
- They help different parts of the problem, but they do not remove the persistent KV bytes. FlashAttention reduces attention-matrix IO and improves kernels. Paged KV allocation reduces fragmentation and packs mixed-length requests better. The stored keys and values still scale with context length and active sequences.
Short definition: what is Cache Arithmetic?