01.08 · Concept
The Context Budget
Calculate KV cache memory from model parameters and context length. Reason about the quadratic prefill cost and why longer contexts change the inference profile.
Context length is a memory and compute budget, not just an API limit. KV cache grows with active tokens across the batch, while prefill attention grows quadratically with prompt length. Longer prompts can therefore reduce concurrency, shift latency towards prefill, and make KV allocation the serving bottleneck.
What this lesson answers
- how to calculate KV cache memory
- why long context makes prefill expensive
- how context length affects inference concurrency
Notes
The context budget is the maximum tokens a serving request can keep “alive” in the model, constrained by KV cache memory and prefill compute: for batch size , sequence length , layers , KV heads , head dim , and bytes per element , the KV cache size is . Prefill attention work scales approximately as , while one-token decode attention scales as .
References
Common questions
- What is the KV cache in LLM inference?
- The KV cache stores attention keys and values for tokens that are still active in a request. During decoding, the model reuses that stored state instead of recomputing it for the whole prefix. The cache lives in GPU memory and grows with batch size, sequence length, layer count, KV heads, head dimension, and element size.
- Why does a longer prompt hurt prefill more than decode?
- Prefill processes the prompt as a sequence and attention compares tokens against other prompt tokens, so its attention work grows quadratically with prompt length. Decode usually adds one token at a time and attends over the existing context, so each generated token grows linearly with the active context.
- How does context length reduce serving throughput?
- The KV cache budget is shared across all active sequences. A single very long request can occupy the same cache capacity as many shorter ones, leaving fewer KV blocks available for admission. Even if memory fits, longer prompts also make prefill heavier, which changes batching, scheduling, and latency behaviour.
Short definition: what is Context Budget?