How do I decide whether to reload or recompute?

Compute both sides for your actual system. Reload cost is the KV footprint divided by effective bandwidth, plus software overhead. Recompute cost is the prefill work for the same prefix divided by sustained prefill throughput. Reload when the first is lower and the copy can be staged without delaying more valuable active work.

Why is this different from normal KV caching?

Normal KV caching usually means keeping keys and values resident so decoding can append one token at a time without reprocessing the whole prefix. Offload and reload adds a memory-tiering decision: the cache may leave accelerator memory while a request is idle, then return later if that exact prefix becomes useful again.

When is offload and reload a bad idea?

It is often poor for tiny prefixes, short idle gaps, non-coalesced cache layouts, or systems where the host link is already on the critical path. It is also wrong if the resumed prompt is not identical, or if adapters, tokenizer behaviour, RoPE settings, layer configuration, or precision policy have changed.

KV Offload and Reload

KV offload and reload is a serving technique that moves a request’s transformer key-value cache out of accelerator memory to a slower memory tier, then copies it back if that prefix is needed again, avoiding prefill recomputation when moving the cached bytes is cheaper than rebuilding them.

The need comes from a mismatch between long-lived agent context and scarce accelerator memory. A model may have to keep weights, active decode state, batches, and many conversation prefixes in HBM at once. If every paused branch keeps its KV cache resident, capacity runs out. If every resumed branch recomputes its prefix, prefill burns accelerator time that could have served new requests.

Mechanically, each generated or prefetched token leaves key and value tensors for every layer. Offload evicts those tensors, often in paged blocks, from HBM to CPU RAM or another lower tier. Reload allocates space, maps or copies the blocks back, and attaches them to the resumed request. The decision is arithmetic: compare KV bytes divided by effective link bandwidth with the FLOPs needed to prefill the same prefix at sustained throughput.

The trade-off is that bandwidth, scheduling, and layout become part of correctness and latency, not just storage details. Large prefixes often favour reload, but small prefixes can be dominated by allocation, pinned memory, page-table work, CUDA graph breaks, NUMA effects, or queueing. It also depends on whether the transfer path is shared with active decode, tool traffic, tensor-parallel communication, or other work that matters more.

Engineers meet KV offload and reload in inference servers that implement paged KV caches, prefix sharing, agent branching, and cluster-level KV routing. It appears as eviction thresholds, host-offload settings, cache-hit paths, and placement policies. A common misunderstanding is that a cache hit is always worth reloading. It is only valid for a byte-identical prefix under the same tokenizer, model state, position handling, precision policy, and adapters.

Go deeper

Inference EngineeringKV Offload and ReloadDo the arithmetic that decides between reloading a cache and recomputing it: bytes to move against the link bandwidth, versus the prefill FLOPs to rebuild the same tokens. Find the prefix length where the answer flips.Open the lesson

Common questions

How do I decide whether to reload or recompute?: Compute both sides for your actual system. Reload cost is the KV footprint divided by effective bandwidth, plus software overhead. Recompute cost is the prefill work for the same prefix divided by sustained prefill throughput. Reload when the first is lower and the copy can be staged without delaying more valuable active work.
Why is this different from normal KV caching?: Normal KV caching usually means keeping keys and values resident so decoding can append one token at a time without reprocessing the whole prefix. Offload and reload adds a memory-tiering decision: the cache may leave accelerator memory while a request is idle, then return later if that exact prefix becomes useful again.
When is offload and reload a bad idea?: It is often poor for tiny prefixes, short idle gaps, non-coalesced cache layouts, or systems where the host link is already on the critical path. It is also wrong if the resumed prompt is not identical, or if adapters, tokenizer behaviour, RoPE settings, layer configuration, or precision policy have changed.

Every term