What is KV cache offload and reload?

KV cache offload means moving a request’s stored attention keys and values out of accelerator memory into a lower memory tier, commonly host RAM. Reload means copying them back if the same prefix is needed again. The point is to avoid running prefill over tokens whose cache has already been produced.

How do you decide whether to reload or recompute a KV cache?

Compare the time to move the KV bytes over the effective link bandwidth with the time to recompute the same prefix using sustained prefill throughput. Reload is attractive when the copy path is faster and does not block more valuable decode work. Recompute is better when copy overhead, contention or invalidation removes the benefit.

When is a reloaded KV cache invalid?

A saved KV cache is only reusable for the exact same prefix under the same model configuration. Tokenisation, system prompt, positional scaling, adapter state, layer structure and precision policy must match. If any of those change, the stored keys and values no longer correspond to the computation being requested, so prefill must run again.

Serving Agents

07.05 · Walkthrough

KV Offload and Reload

Do the arithmetic that decides between reloading a cache and recomputing it: bytes to move against the link bandwidth, versus the prefill FLOPs to rebuild the same tokens. Find the prefix length where the answer flips.

KV reload wins when moving the saved keys and values over the available link is cheaper than rebuilding them with prefill compute. The decision is a bandwidth versus FLOPs calculation, with practical thresholds for small prefixes where allocation, paging, scheduling and contention can dominate the raw copy time.

What this lesson answers

when should KV cache be reloaded instead of recomputed
how to calculate KV cache offload reload time
why small prefixes may not be worth KV offload

Notes

KV offload/reload means evicting a request’s KV cache from accelerator memory to a lower tier, then restoring it if the conversation or agent branch is resumed, instead of recomputing those keys and values by running prefill over the same prefix. For a prefix of $L$ tokens, transformer layers $N$ , KV heads $H_{k v}$ , head dimension $d$ , and element size $b$ bytes, the KV footprint is $B_{K V} = 2 N L H_{k v} d b$ , where the factor 2 is key plus value. Reload time over an effective host/GPU or fabric bandwidth $β$ is $t_{r e l o a d} = B_{K V} / β$ plus allocation and page-table overhead.

References

Common questions

What is KV cache offload and reload?: KV cache offload means moving a request’s stored attention keys and values out of accelerator memory into a lower memory tier, commonly host RAM. Reload means copying them back if the same prefix is needed again. The point is to avoid running prefill over tokens whose cache has already been produced.
How do you decide whether to reload or recompute a KV cache?: Compare the time to move the KV bytes over the effective link bandwidth with the time to recompute the same prefix using sustained prefill throughput. Reload is attractive when the copy path is faster and does not block more valuable decode work. Recompute is better when copy overhead, contention or invalidation removes the benefit.
When is a reloaded KV cache invalid?: A saved KV cache is only reusable for the exact same prefix under the same model configuration. Tokenisation, system prompt, positional scaling, adapter state, layer structure and precision policy must match. If any of those change, the stored keys and values no longer correspond to the computation being requested, so prefill must run again.

Short definition: what is KV Offload and Reload?

07.04 · Serving AgentsThe KV Memory Hierarchy 07.06 · Serving AgentsTool Calls in the Loop