Why can’t the prefill worker send only the next-token logits?

The logits are only enough to choose the next token. To generate later tokens, decode must repeatedly attend over the original prompt and all generated tokens so far. The prompt part of that attention state is the KV cache built during prefill, so without moving or sharing it the decode worker would have to recompute prefill.

When is moving the KV cache worth it?

It is worth it when the benefit from separating prefill and decode queues exceeds the time to transfer, register, schedule, and possibly adapt the cache layout. Long prefills, busy prefill workers, idle nearby decode workers, compatible block formats, and NVLink-like topology help. Short prompts, tiny output budgets, and congested cross-node links hurt.

What does the transfer layer do besides copying bytes?

It maps cache blocks from the prefill engine into a form the decode engine can adopt. That includes choosing a transport, using registered GPU memory where possible, preserving block tables, avoiding host staging, handling prefix-cache reuse, and coordinating ownership. If it must repack or transpose the cache, the copy can become a bandwidth-bound computation.

Is RDMA enough to make cross-node prefill and decode efficient?

Not by itself. RDMA helps only if GPU memory is reachable efficiently, buffers are registered, NIC placement is sensible, and the receiver can consume the same KV layout. Otherwise the path may silently involve host staging, extra PCIe traffic, setup overhead, or repacking. The measured effective bandwidth matters more than the advertised link.

Moving the KV Cache

Moving the KV cache is the handoff of a request’s stored attention keys and values from a prefill worker to a decode worker in disaggregated LLM serving. It is the critical cost because decode cannot continue from logits alone; it must receive the prompt’s per-layer KV state in a usable layout.

Disaggregated serving tries to place the prompt-heavy prefill phase and the token-by-token decode phase on different workers, because they stress hardware differently and interfere in one shared queue. The catch is that prefill produces a large KV cache, not just a small result. Decode needs that cache to attend back over the prompt, so the request cannot simply be routed by sending an id and a few logits.

Concretely, the prefill worker finishes the prompt pass, then the serving system transfers the KV blocks for every layer to memory visible to the chosen decode worker. On one machine this may be a GPU-to-GPU copy over NVLink or NVSwitch. Across machines it usually means RDMA, registered GPU memory, NIC placement, and a protocol that tells the receiver which cache blocks now belong to the request.

The trade-off is latency and bandwidth at the exact point where first-token latency matters. If the cache move takes longer than the queueing benefit of separating prefill from decode, disaggregation is a regression. The honest answer is topology-dependent: NVLink can make the handoff cheap enough, while cross-node RDMA can be dominated by registration, scheduling, PCIe effects, congestion, or layout conversion.

Engineers meet this in vLLM, SGLang, TensorRT-LLM, Dynamo-style serving stacks, and cluster schedulers that split prefill and decode pools. The transfer layer must understand paged or block-based KV layouts, prefix-cache hits, memory registration, and worker topology. It is commonly misunderstood as a networking problem; it is really a scheduling, memory-layout, and transport problem at once.

Go deeper

Inference EngineeringMoving the KV CacheExplain why the KV transfer between a prefill worker and a decode worker is the engineering crux of disaggregation, and how NVLink, RDMA and a transfer layer decide whether the split is a win or a regression.Open the lesson

Common questions

Why can’t the prefill worker send only the next-token logits?: The logits are only enough to choose the next token. To generate later tokens, decode must repeatedly attend over the original prompt and all generated tokens so far. The prompt part of that attention state is the KV cache built during prefill, so without moving or sharing it the decode worker would have to recompute prefill.
When is moving the KV cache worth it?: It is worth it when the benefit from separating prefill and decode queues exceeds the time to transfer, register, schedule, and possibly adapt the cache layout. Long prefills, busy prefill workers, idle nearby decode workers, compatible block formats, and NVLink-like topology help. Short prompts, tiny output budgets, and congested cross-node links hurt.
What does the transfer layer do besides copying bytes?: It maps cache blocks from the prefill engine into a form the decode engine can adopt. That includes choosing a transport, using registered GPU memory where possible, preserving block tables, avoiding host staging, handling prefix-cache reuse, and coordinating ownership. If it must repack or transpose the cache, the copy can become a bandwidth-bound computation.
Is RDMA enough to make cross-node prefill and decode efficient?: Not by itself. RDMA helps only if GPU memory is reachable efficiently, buffers are registered, NIC placement is sensible, and the receiver can consume the same KV layout. Otherwise the path may silently involve host staging, extra PCIe traffic, setup overhead, or repacking. The measured effective bandwidth matters more than the advertised link.

Every term