Is the context budget the same as the model's maximum context window?

No. The model's maximum context window is the architectural or configured limit it can address. The serving context budget is the amount your hardware and scheduler can keep live at once. A model may support a long window, while the service cannot admit many such requests without exhausting KV cache or delaying prefill.

Why does a longer prompt hurt latency more than a longer generation?

Prefill compares prompt tokens with other prompt tokens, so attention work grows roughly with the square of prompt length. Decode usually adds one token at a time and attends back over the existing cache, so each step grows roughly linearly with live context. For long prompts, prefill can dominate time to first token.

What is actually stored in the KV cache?

For each processed token, the model stores the key and value tensors produced by attention layers. Future generated tokens reuse those tensors instead of recomputing the whole history. The cache therefore grows with active tokens, layers, KV heads, head dimension, and numeric precision. Query tensors are computed for the current step and are not cached in the same way.

How do engineers increase the usable context budget?

They can reduce batch pressure, reserve more GPU memory for KV, use attention variants with fewer KV heads, use a smaller KV dtype if quality and kernels allow it, or adopt paged KV allocation to reduce fragmentation. They can also limit retrieved text, summarise history, or use sliding-window attention when the model supports it.

Context Budget

A context budget is the maximum number of tokens an inference service can keep active for a model request or batch, limited mainly by KV cache memory and prompt prefill compute. It determines how much prompt, history, retrieval content, and generated output can coexist before the server must reject, truncate, or evict work.

Long context is not just a product setting. Every active token leaves data behind in the attention cache, and long prompts must be processed before the first generated token appears. That means increasing the allowed context can reduce concurrency, increase latency, or exhaust GPU memory even when the model weights fit comfortably. The common misunderstanding is to treat context length as a model-only capability rather than a serving capacity shared across active requests.

During prefill, the model reads the prompt and writes key and value tensors for each layer into the KV cache. The cache size is batch size times sequence length times layers times keys and values times KV heads times head dimension times bytes per element. Attention during prefill grows roughly with sequence length squared, while each later decode step attends over the existing cache and grows roughly linearly with the live sequence length.

The trade-off is direct: more context gives the model more evidence, but spends memory and changes the latency profile. Grouped-query or multi-query attention reduces KV heads, lower-precision KV reduces bytes per element, and paged allocation reduces waste, but none makes long context free. Sliding windows and chunked attention can cap the effective history, but only if the model and serving stack are designed for that behaviour.

Engineers meet the context budget in scheduler admission control, KV block allocation, batch sizing, and max-context configuration. A serving system must decide whether a new sequence group can fit into available KV blocks, whether prefill should run now, and how many decode requests can share the GPU. The honest sizing answer is always workload-dependent: prompt length distribution, generation length, batching policy, KV dtype, attention variant, and reserved runtime memory all matter.

Go deeper

Inference EngineeringThe Context BudgetCalculate KV cache memory from model parameters and context length. Reason about the quadratic prefill cost and why longer contexts change the inference profile.Open the lesson

Common questions

Is the context budget the same as the model's maximum context window?: No. The model's maximum context window is the architectural or configured limit it can address. The serving context budget is the amount your hardware and scheduler can keep live at once. A model may support a long window, while the service cannot admit many such requests without exhausting KV cache or delaying prefill.
Why does a longer prompt hurt latency more than a longer generation?: Prefill compares prompt tokens with other prompt tokens, so attention work grows roughly with the square of prompt length. Decode usually adds one token at a time and attends back over the existing cache, so each step grows roughly linearly with live context. For long prompts, prefill can dominate time to first token.
What is actually stored in the KV cache?: For each processed token, the model stores the key and value tensors produced by attention layers. Future generated tokens reuse those tensors instead of recomputing the whole history. The cache therefore grows with active tokens, layers, KV heads, head dimension, and numeric precision. Query tensors are computed for the current step and are not cached in the same way.
How do engineers increase the usable context budget?: They can reduce batch pressure, reserve more GPU memory for KV, use attention variants with fewer KV heads, use a smaller KV dtype if quality and kernels allow it, or adopt paged KV allocation to reduce fragmentation. They can also limit retrieved text, summarise history, or use sliding-window attention when the model supports it.

Every term