04.06 · Concept
Prefill/Decode Interference
Show why one GPU doing both phases does neither well: prefill saturates compute, decode saturates bandwidth, and mixing them means each phase pays the other's stall.
Prefill/decode interference happens when prompt processing and token generation share a GPU despite having opposite bottlenecks. Prefill wants large compute-heavy matrix work; decode wants bandwidth-heavy token steps. Mixed scheduling makes decode wait behind long compute kernels, while prefill loses efficiency to small memory-bound work and scheduling gaps.
What this lesson answers
- why prefill and decode interfere on one GPU
- when should prefill decode be disaggregated
- how chunked prefill affects token latency
Notes
Prefill/decode interference is the loss that occurs when one GPU schedules prompt processing and token generation in the same execution stream even though the two phases sit on different sides of the roofline. For an operation with arithmetic intensity , attainable throughput is . Prefill runs large matrix-matrix products over prompt tokens, so its QKV, MLP, and output projections have high reuse and push toward .
Common questions
- Why does prefill behave differently from decode?
- Prefill processes many prompt tokens at once, so projection weights are reused across a large batch of work and tensor cores stay busy. Decode usually advances each sequence by a single token, so it repeatedly streams weights and KV-cache data with much less reuse. That puts prefill closer to compute-bound execution and decode closer to memory-bandwidth-bound execution.
- Why does continuous batching make the problem visible?
- Continuous batching keeps admitting new work while existing requests are generating tokens. That improves utilisation, but it also places prompt prefill and decode steps into the same scheduling path. Long prefill kernels can delay next-token delivery, while protecting decode by cutting prefill into small pieces can make prefill much less efficient.
- When is separating prefill and decode worth it?
- Separation helps when prompts or continuations are large enough that avoiding repeated interference outweighs KV transfer, queueing, and worker coordination costs. It is less useful for short prompts, very short generations, or already homogeneous traffic. Chunked prefill is the middle ground: it limits decode stalls without fully moving phases to separate workers.
Short definition: what is Prefill/Decode Interference?