Why is the KV cache moved instead of just sending logits?

Decode needs the prompt’s keys and values for attention at every generated token. Logits from prefill are not enough, because the decode worker must attend over the whole prompt state repeatedly. That makes the handoff a large tensor ownership transfer, not a small API result.

Why is NVLink usually easier than RDMA for this split?

NVLink can keep the transfer inside the GPU fabric and avoid host staging when the workers are on the same suitable topology. RDMA can work across nodes, but it depends on GPUDirect, memory registration, NIC placement and protocol design. If those fail, the cache may bounce through CPUs and PCIe.

When should prefill and decode stay on the same worker?

Keep them together when the prompt is small, the expected output is short, the network path is congested, or the decode worker cannot import the cache layout directly. In those cases, moving and repacking KV can cost more than the scheduling benefit from separating compute-heavy prefill and memory-heavy decode.

The Batch

04.08 · Walkthrough

Moving the KV Cache

Explain why the KV transfer between a prefill worker and a decode worker is the engineering crux of disaggregation, and how NVLink, RDMA and a transfer layer decide whether the split is a win or a regression.

KV cache movement is the hard boundary in disaggregated LLM serving: prefill produces layer-wise keys and values that decode must already have. The split only helps when queueing and occupancy gains outweigh transfer, registration, layout conversion and scheduling costs across NVLink, RDMA or a serving-specific transfer layer.

What this lesson answers

why move KV cache between prefill and decode
when does disaggregated LLM serving become slower
NVLink versus RDMA for KV cache transfer

Notes

Disaggregated serving splits the request lifecycle so a prefill worker runs the prompt-wide matrix-heavy pass and a decode worker runs the token-by-token memory-bound pass; the handoff is not logits but the completed KV cache for every layer, because decode needs those keys and values to attend over the prompt. The transfer size for one sequence is $B_{KV} = L \cdot T \cdot 2 \cdot H_{k v} \cdot d_{h} \cdot s$ , where $L$ is layer count, $T$ prompt tokens, the factor $2$ is K and V, $H_{k v}$ is KV-head count under MHA/MQA/GQA, $d_{h}$ is head dimension, and $s$ is bytes per element.

References

Common questions

Why is the KV cache moved instead of just sending logits?: Decode needs the prompt’s keys and values for attention at every generated token. Logits from prefill are not enough, because the decode worker must attend over the whole prompt state repeatedly. That makes the handoff a large tensor ownership transfer, not a small API result.
Why is NVLink usually easier than RDMA for this split?: NVLink can keep the transfer inside the GPU fabric and avoid host staging when the workers are on the same suitable topology. RDMA can work across nodes, but it depends on GPUDirect, memory registration, NIC placement and protocol design. If those fail, the cache may bounce through CPUs and PCIe.
When should prefill and decode stay on the same worker?: Keep them together when the prompt is small, the expected output is short, the network path is congested, or the decode worker cannot import the cache layout directly. In those cases, moving and repacking KV can cost more than the scheduling benefit from separating compute-heavy prefill and memory-heavy decode.

Short definition: what is Moving the KV Cache?

04.07 · The BatchDisaggregation 04.09 · The BatchTTFT, TPOT, ITL