Why not keep the whole KV cache in GPU memory?

GPU HBM is the scarce tier needed by the currently decoding batch. Long agent sessions may pause during tool calls, retrieval, or code execution while their prefixes remain useful. Keeping every inactive prefix hot reduces batching capacity and competes with weights, activations, graphs, and allocator overhead.

Is loading KV from storage better than recomputing the prompt?

It depends on the model, prefix length, and storage tier. For large models and long prefixes, copying saved KV from host memory, NVMe, or a pooled store can be cheaper than running prefill again. For short prompts or small models, recomputation can beat a remote fetch.

What makes a KV memory hierarchy work in practice?

The cache must be split into manageable blocks, with a scheduler that knows which blocks are needed before attention runs. Systems such as paged KV caches and prefix-sharing structures make placement granular enough to move, reuse, evict, and reload prefixes without treating each sequence as one monolithic tensor.

Serving Agents

07.04 · Concept

The KV Memory Hierarchy

Place the KV cache on a storage hierarchy: HBM, host DRAM, local NVMe, a remote pooled store. Explain why production fleets now hold more cache off the GPU than on it. This is module 2's memory hierarchy, one level up.

KV cache belongs on a memory hierarchy, not permanently on the GPU. Active attention blocks need HBM, but idle agent prefixes can sit in host DRAM, local NVMe, or a remote pooled store. The trade is I/O latency against avoided prefill, with block placement and scheduling deciding whether reuse pays.

What this lesson answers

where should KV cache live in production inference
why store KV cache off GPU memory
when does KV cache offloading hurt latency

Notes

The KV memory hierarchy is the policy and mechanism for placing per-token attention state across GPU HBM, host DRAM, local NVMe, and remote pooled memory instead of treating the KV cache as an all-or-nothing tensor resident on the serving GPU. For a decoder-only model, the bytes of KV for one sequence are $B_{KV} = 2 \cdot L \cdot T \cdot H_{kv} \cdot D \cdot b$ , where $2$ is key plus value, $L$ is layer count, $T$ is cached tokens, $H_{kv}$ is the number of KV heads, $D$ is head dimension, and $b$ is bytes per element.

References

Common questions

Why not keep the whole KV cache in GPU memory?: GPU HBM is the scarce tier needed by the currently decoding batch. Long agent sessions may pause during tool calls, retrieval, or code execution while their prefixes remain useful. Keeping every inactive prefix hot reduces batching capacity and competes with weights, activations, graphs, and allocator overhead.
Is loading KV from storage better than recomputing the prompt?: It depends on the model, prefix length, and storage tier. For large models and long prefixes, copying saved KV from host memory, NVMe, or a pooled store can be cheaper than running prefill again. For short prompts or small models, recomputation can beat a remote fetch.
What makes a KV memory hierarchy work in practice?: The cache must be split into manageable blocks, with a scheduler that knows which blocks are needed before attention runs. Systems such as paged KV caches and prefix-sharing structures make placement granular enough to move, reuse, evict, and reload prefixes without treating each sequence as one monolithic tensor.

Short definition: what is KV Memory Hierarchy?

07.03 · Serving AgentsCache-Aware Routing 07.05 · Serving AgentsKV Offload and Reload