04.07 · Concept
Disaggregation
Describe running prefill and decode on separate GPU pools so each scales independently, and state honestly when it does not pay: short prompts, low concurrency, or fewer GPUs than it takes to keep both pools busy.
Disaggregated inference runs prompt prefill and token decode on separate GPU pools, passing the KV cache between them. It helps when the two phases need different batch shapes and enough traffic exists to keep both pools productive. It can hurt with short prompts, low concurrency, weak networking, or too few GPUs.
What this lesson answers
- when should prefill and decode use separate GPUs
- why does disaggregated inference need KV cache transfer
- when does prefill decode disaggregation hurt latency
Notes
Disaggregation is the serving mechanism that runs the prefill phase and the decode phase of the same request on different GPU pools, with the prefill workers computing the prompt KV cache and the decode workers consuming that KV cache to emit one token at a time. The split is useful because prefill is compute-heavy matrix multiplication over many prompt tokens, while decode is usually memory-bandwidth-heavy weight streaming plus KV reads for a single new token per sequence.
References
Common questions
- What is prefill decode disaggregation in LLM serving?
- It is an inference architecture where prompt processing runs on one GPU pool and autoregressive token generation runs on another. The prefill side builds the KV cache for the input prompt, then the decode side receives that cache and generates output tokens. The point is to size and batch the two phases independently.
- Why split prefill and decode instead of using one scheduler?
- Prefill and decode stress the hardware differently. Prefill benefits from large token batches and compute-heavy matrix work, especially for long prompts. Decode benefits from many live sequences because each step is dominated by repeatedly reading model weights and KV state. A single scheduler often compromises between those batch shapes.
- When is disaggregated serving a bad idea?
- It is usually a poor fit when prompts are short, request concurrency is low, or the cluster is not large enough to keep separate pools busy. The KV cache handoff, extra scheduling, network cost, and loss of local batching can exceed any gain from independent scaling.
Short definition: what is Disaggregation?