Is disaggregation the same as pipeline parallelism?

No. Pipeline parallelism splits model layers or stages within a forward pass and passes activations between devices. Disaggregation splits two temporal phases of a request: prompt prefill first, decode afterwards. The important object handed across is the KV cache for the prompt, not an activation microbatch flowing through layer partitions.

When does disaggregation usually help?

It tends to help when prompts are long, generations are long enough to make decode substantial, and concurrency is high enough for both pools to form useful batches. In that regime, prefill GPUs can be sized for bursty compute-heavy work, while decode GPUs can be sized for steady memory-bandwidth-heavy token generation.

When should I avoid disaggregated prefill and decode?

Avoid assuming it helps for short chat turns, sparse traffic, or a small GPU fleet. In those cases the KV transfer and extra scheduling path may cost more than the queueing improvement. A unified worker with continuous batching can be simpler and faster if it already keeps the hardware busy.

Why is KV cache movement central to the design?

The decode worker cannot continue a sequence from the prompt text alone without recomputing prefill. It needs the prompt’s keys and values for every relevant layer so future tokens can attend to prior context. Disaggregation is practical only when those KV blocks can be represented, transferred, and installed efficiently enough.

Disaggregation

Disaggregation is an inference-serving architecture that runs prompt prefill and token decode on separate GPU pools. Prefill workers compute the prompt’s KV cache, transfer it to decode workers, and decode workers then generate output tokens, letting each phase be batched, queued, and scaled for its own bottleneck.

Disaggregation exists because prefill and decode stress hardware in different ways. Prefill processes many prompt tokens at once and is usually dominated by large matrix multiplications. Decode advances each live sequence one token at a time and often bottlenecks on reading model weights and KV cache from memory. A single unified scheduler must compromise between these shapes, so one phase can leave GPUs poorly utilised while waiting for the other phase’s preferred batch.

In a disaggregated server, an incoming request is routed first to a prefill worker. That worker runs the prompt through the model and produces the KV cache for the sequence. The system then sends that cache, plus request metadata, to a decode worker. The decode pool keeps many active sequences resident and repeatedly schedules decode steps, consuming the transferred KV cache and appending new KV entries as tokens are emitted.

The trade-off is that the phase boundary is not free. KV cache can be large for long prompts, and moving it adds network, serialisation, scheduling, and cache-locality costs. Disaggregation helps only when better queueing and independent pool sizing outweigh those costs. It commonly does not pay for short prompts, low concurrency, weak fabrics, or clusters with too few GPUs to keep both pools busy.

Engineers meet disaggregation in production LLM serving stacks that expose prefill workers, decode workers, paged KV cache, and routing between them. It is often discussed alongside continuous batching, prefix caching, tensor parallelism, and speculative decoding. A common misunderstanding is to treat it as a faster kernel technique. It is really a serving layout and scheduling decision, and whether it wins depends on workload mix.

Go deeper

Inference EngineeringDisaggregationDescribe running prefill and decode on separate GPU pools so each scales independently, and state honestly when it does not pay: short prompts, low concurrency, or fewer GPUs than it takes to keep both pools busy.Open the lesson

Common questions

Is disaggregation the same as pipeline parallelism?: No. Pipeline parallelism splits model layers or stages within a forward pass and passes activations between devices. Disaggregation splits two temporal phases of a request: prompt prefill first, decode afterwards. The important object handed across is the KV cache for the prompt, not an activation microbatch flowing through layer partitions.
When does disaggregation usually help?: It tends to help when prompts are long, generations are long enough to make decode substantial, and concurrency is high enough for both pools to form useful batches. In that regime, prefill GPUs can be sized for bursty compute-heavy work, while decode GPUs can be sized for steady memory-bandwidth-heavy token generation.
When should I avoid disaggregated prefill and decode?: Avoid assuming it helps for short chat turns, sparse traffic, or a small GPU fleet. In those cases the KV transfer and extra scheduling path may cost more than the queueing improvement. A unified worker with continuous batching can be simpler and faster if it already keeps the hardware busy.
Why is KV cache movement central to the design?: The decode worker cannot continue a sequence from the prompt text alone without recomputing prefill. It needs the prompt’s keys and values for every relevant layer so future tokens can attend to prior context. Disaggregation is practical only when those KV blocks can be represented, transferred, and installed efficiently enough.

Every term