What Is Actually Stored

Notes

A KV cache is the per-layer, per-sequence store of the key and value vectors produced for tokens that have already been prefilling or decoded; it deliberately does not store queries. For one attention head at layer $l$, with hidden state $x_t^{(l)}$, the projections are $q_t=x_t{W}_Q$, $k_t=x_t{W}_K$, and $v_t=x_t{W}_V$, and causal attention for the newly generated token is $o_t=\mathrm{softmax}(q_t{K}_{\le t}^{\top }/\sqrt{d_h})V_{\le t}$. During autoregressive decoding, $K_{\le t-1}$ and $V_{\le t-1}$ are reused for every future token, while $q_t$ is only needed to ask one question of the past at the current position. Thus the cache contains $K$ and $V$ indexed by layer, KV head, token position, and head dimension; the current token’s $Q$ is recomputed from the current hidden state and discarded after the attention output is formed. FlashAttention, from Dao et al. 2022 and FlashAttention-2 in 2023, changes how this attention is tiled and accumulated, but not the semantic fact that the persistent autoregressive state is past keys and values, not past queries.

Tracing a single grouped-query attention layer makes the storage boundary explicit. Suppose a model has hidden size $8192$, $64$ query heads, $8$ KV heads, and head dimension $128$, as in a realistic Llama-family 70B-style configuration. At position $t$, the layer computes $Q_t$ with shape $64\times 128$, $K_t$ with shape $8\times 128$, and $V_t$ with shape $8\times 128$. The $64$ query heads are partitioned into groups sharing the $8$ KV heads; each query head attends to one of the KV heads’ history. The implementation appends only $K_t$ and $V_t$ to the cache, so the per-layer token addition is $2\cdot 8\cdot 128$ elements, not $(64+8+8)\cdot 128$. On the next decode step, $Q_{t+1}$ is newly projected from the new residual stream, while $K_{\le t}$ and $V_{\le t}$ are read from cache and concatenated logically with $K_{t+1},V_{t+1}$. PagedAttention in vLLM, from Kwon et al. 2023, virtualizes this exact append-and-read pattern into block tables rather than contiguous per-request tensors.

The arithmetic shows why engineers obsess over KV and not Q. For a 70B model in FP16 with $80$ decoder layers, $8$ KV heads, and $d_h=128$, each token’s KV cache is $L\cdot 2\cdot h_{kv}\cdot d_h\cdot 2$ bytes $=80\cdot 2\cdot 8\cdot 128\cdot 2=327,680$ bytes, about $320$ KiB per token per sequence. A $4096$-token conversation therefore consumes $327,680\cdot 4096=1,342,177,280$ bytes, about $1.25$ GiB, before allocator fragmentation and metadata. On an 80GB H100 with $3.35$ TB/s HBM bandwidth, reading the full KV history for one new token at length $4096$ moves roughly $1.34$ GB, so the bandwidth lower bound is $1.34/3350\approx 0.00040$ s, or $0.40$ ms per generated token for KV reads alone, ignoring matmuls, softmax, and interconnect. Storing past queries would add $80\cdot 64\cdot 128\cdot 2=1,310,720$ bytes per token, four times the KV footprint, yet those old queries are never multiplied by future keys in causal decoding.

Prefill and decode differ only in lifetime, not in projection identity. During prefill, the system processes many prompt tokens together, so it computes $Q,K,V$ for all prompt positions and can use FlashAttention kernels to avoid materializing the full attention matrix. After the layer’s prefill attention is complete, the $Q$ block has no future use and is freed, while the generated $K$ and $V$ blocks become the initial cache for decode. During decode, each step has a batch of one logical position per active sequence, so the kernel computes fresh $Q_t,K_t,V_t$, writes $K_t,V_t$ into the cache, and performs attention from $Q_t$ over cached $K,V$. TensorRT-LLM’s inflight batching and paged KV cache, SGLang’s RadixAttention for prefix reuse, NVIDIA Dynamo’s disaggregated serving runtime, and llm-d’s Kubernetes-native inference stack all build scheduling and memory movement around this invariant: the reusable state is KV pages or blocks. EAGLE, from Li et al. 2024, speculates future tokens with a draft mechanism, but accepted tokens still contribute only their keys and values to the target model’s cache.

The technique stops helping when the cost of preserving and fetching KV exceeds the cost it saves or when the cached tensors are not valid for the computation being requested. At extremely short contexts, say $t<16$, cache management overhead, page-table indirection, and kernel launch structure can dominate the small avoided projection work, so a fused no-cache path may be faster in microbenchmarks. Under severe memory pressure, a $32$k-token request for the same 70B example needs $327,680\cdot 32,768\approx 10.74$ GB of KV for one sequence, so seven such sequences nearly fill an 80GB H100 before weights if weights are not tensor-parallel sharded elsewhere; paging to CPU or NVMe can make time-to-token worse than recomputing or truncating. The cache is also invalid across weight changes, LoRA adapter changes that affect $W_K$ or $W_V$, different RoPE scaling, different prompt bytes, or different layer normalization states. Prefix caching in vLLM, SGLang, TensorRT-LLM, Dynamo, or llm-d is therefore exact-match or carefully hashed reuse, not semantic similarity reuse.

A correct mental model is that a cached token is represented in every layer by two projected memory vectors, not by its text embedding and not by its query. The next token’s hidden state flows through $W_Q$ to form a query that scores all older keys, and the resulting probabilities mix older values; after that, the query has served its only causal purpose. This is why multi-query and grouped-query attention reduce serving memory by shrinking $h_{kv}$ while leaving $h_q$ large for model quality, and why cache quantization targets $K,V$ tensors rather than $Q$. It is also why PagedAttention’s block allocator, FlashAttention’s IO-aware attention, TensorRT-LLM’s paged-context kernels, SGLang’s prefix-radix cache, Dynamo’s KV-routing work, llm-d’s distributed KV-aware deployment, and EAGLE’s speculative acceptance path all talk about KV residency, KV transfer, and KV reuse. If a trace shows old queries being stored persistently during standard causal decoding, that trace is either naming a temporary kernel tile “Q cache” imprecisely or implementing a different algorithm than ordinary transformer inference.