Is KV cache on remote storage faster than CPU memory or local SSD?

Usually no. Remote pooled storage is normally slower and less predictable than host DRAM or local NVMe. Its value is capacity, sharing, and surviving worker placement changes, not raw access speed. It becomes useful when compression, prefix reuse, or avoiding a large prefill outweighs the network and scheduling cost.

Why not just recompute the prompt instead of storing KV?

It depends on model size, prefix length, load, and reuse probability. Recomputing uses GPU compute and HBM bandwidth during prefill, which may be the scarce resource. Restoring KV uses I/O bandwidth and memory capacity. For long prefixes on large models, copying cached KV is often cheaper; for short or rarely reused prefixes, recomputation can be simpler and faster.

Is the KV memory hierarchy the same as PagedAttention?

No. PagedAttention is one important mechanism: it divides KV into fixed-size blocks and maps logical sequence positions to physical storage. A KV memory hierarchy is broader. It includes placement across HBM, DRAM, NVMe, and remote stores, plus scheduling, eviction, compression, prefix sharing, and transfer decisions.

KV Memory Hierarchy

A KV memory hierarchy is the serving-system design that keeps attention key-value cache across GPU HBM, host DRAM, local NVMe, and remote pooled storage, moving blocks between tiers so active tokens are near the GPU while idle but reusable conversation state does not consume scarce accelerator memory.

The need comes from long-running inference, especially agents. A chat, tool loop, browser task, or code run may pause for external work and then resume with a large prefix that is expensive to recompute. Keeping every such prefix in GPU memory would crowd out active batches. Throwing it away wastes prefill work. The hierarchy is the compromise: retain valuable KV state, but reserve HBM for sequences being attended right now.

Concretely, the KV cache is split into blocks rather than treated as one tensor. The scheduler tracks which logical token ranges live in which physical blocks and which tier contains each block. Before an attention kernel runs, the required key and value blocks must be present in HBM. Warm blocks may be copied from host DRAM, colder blocks from NVMe, and fleet-level blocks from a remote store, often compressed or deduplicated by prefix.

The trade-off is that misses become latency. HBM is fast because it is next to the GPU; every lower tier adds transfer time, bandwidth contention, bookkeeping, and failure modes. Offloading only pays when copying saved KV is cheaper than recomputing the prefix and when reuse is likely. It can hurt short prompts, low-resumption workloads, tiny models, or systems whose active working set thrashes between GPU and storage.

Engineers meet this in inference servers through paged KV caches, prefix caching, disaggregated prefill and decode, and cache eviction policy. vLLM-style paging, SGLang-style prefix sharing, TensorRT-LLM KV reuse, and newer distributed serving stacks all expose the same core problem: deciding what KV belongs on the GPU, what can wait elsewhere, and when moving it is worth more than recomputing it.

Go deeper

Inference EngineeringThe KV Memory HierarchyPlace the KV cache on a storage hierarchy: HBM, host DRAM, local NVMe, a remote pooled store. Explain why production fleets now hold more cache off the GPU than on it. This is module 2's memory hierarchy, one level up.Open the lesson

Common questions

Is KV cache on remote storage faster than CPU memory or local SSD?: Usually no. Remote pooled storage is normally slower and less predictable than host DRAM or local NVMe. Its value is capacity, sharing, and surviving worker placement changes, not raw access speed. It becomes useful when compression, prefix reuse, or avoiding a large prefill outweighs the network and scheduling cost.
Why not just recompute the prompt instead of storing KV?: It depends on model size, prefix length, load, and reuse probability. Recomputing uses GPU compute and HBM bandwidth during prefill, which may be the scarce resource. Restoring KV uses I/O bandwidth and memory capacity. For long prefixes on large models, copying cached KV is often cheaper; for short or rarely reused prefixes, recomputation can be simpler and faster.
Is the KV memory hierarchy the same as PagedAttention?: No. PagedAttention is one important mechanism: it divides KV into fixed-size blocks and maps logical sequence positions to physical storage. A KV memory hierarchy is broader. It includes placement across HBM, DRAM, NVMe, and remote stores, plus scheduling, eviction, compression, prefix sharing, and transfer decisions.

Every term