Prefill/Decode Interference
Prefill/decode interference is the throughput and latency loss caused when prompt processing and token-by-token generation share the same GPU scheduling path. Prefill is compute-heavy, with large dense matrix work, while decode is usually memory-bandwidth-heavy, so colocating them makes prefill wait on memory-shaped stalls and decode wait behind long compute kernels.
The problem appears because LLM inference is not one uniform workload. A request first runs prefill, which processes the whole prompt and builds the KV cache. It then runs decode, which repeatedly produces the next token using that cache. Continuous batching improves utilisation by mixing requests at different stages, but it also puts two phases with very different bottlenecks into the same execution stream.
Mechanically, prefill runs large matrix-matrix operations across many prompt tokens, so weights are reused and tensor cores can stay busy. Decode handles one new token per active sequence, so it repeatedly streams model weights and reads KV-cache blocks with much less reuse. When scheduled together, decode can sit behind prefill’s long kernels, while prefill is chopped up or delayed by decode-friendly scheduling.
The trade-off is not simply that sharing a GPU is bad. Chunking prefill can protect next-token latency, but smaller chunks reduce GEMM efficiency and add scheduling overhead. Separating prefill and decode onto different workers avoids much of the contention, but then the system must transfer KV cache, route requests, manage queues, and decide when the handoff cost is worth paying.
Engineers meet this in inference servers that support continuous or in-flight batching, paged KV caches, long prompts, and latency SLOs. It shows up as time-to-first-token and inter-token latency spikes when large prompts arrive beside active generations. Common mitigations include chunked prefill, priority scheduling for decode, prefix reuse, and full prefill/decode disaggregation with a KV-cache transfer between worker pools.
Common questions
- Is prefill/decode interference just GPU contention?
- No. Ordinary contention means two jobs compete for the same device. Prefill/decode interference is more specific: the jobs stress different parts of the roofline. Prefill wants long, dense compute kernels with high reuse. Decode wants low-latency steps dominated by memory movement. A scheduler that mixes them makes each phase inherit delays from the other phase’s bottleneck.
- Why does continuous batching make this problem visible?
- Continuous batching keeps the GPU busy by admitting new work while older requests are still decoding. That is good for utilisation, but it means a long prompt prefill can enter the same batch as active decode requests. The decode requests then wait for prefill kernels, or the prefill is sliced smaller to preserve latency, which reduces its efficiency.
- Does disaggregating prefill and decode always help?
- No. It depends on prompt length, generation length, batch shape, network or fabric cost, and the latency target. Disaggregation helps when the avoided interference is larger than the cost of routing the request and moving the KV cache. For short prompts or very short generations, a colocated schedule can be faster and simpler.
- How is chunked prefill different from disaggregation?
- Chunked prefill keeps both phases on the same worker but limits how long prefill can occupy the GPU before decode gets another turn. Disaggregation sends prefill to one pool and decode to another, then transfers the computed KV cache. Chunking is simpler but still mixes bottlenecks; disaggregation is cleaner but adds data movement and orchestration.