06.03 · Concept
Pipeline Parallelism
Split a model by layer across devices, describe the bubble that idle stages create, and say when PP is the right answer across nodes where TP's collectives are too expensive.
Pipeline parallelism splits a model’s layers across devices, passing activations between adjacent stages instead of synchronising every layer. It is useful when weights or KV cache exceed one device, or when tensor-parallel collectives are too costly across nodes, but it introduces idle fill and drain time called the pipeline bubble.
What this lesson answers
- when should I use pipeline parallelism for inference
- what is the pipeline bubble in model serving
- pipeline parallelism versus tensor parallelism across nodes
Notes
Pipeline parallelism for LLM inference partitions the transformer block stack by depth, assigning contiguous layer ranges to devices or nodes, so stage computes and sends activations, not weights, to stage . For a batch split into microbatches over stages with roughly equal stage time , the classic GPipe schedule has latency and useful work , so utilization is ; the idle fraction is the pipeline bubble.
Common questions
- Why does pipeline parallelism create idle GPU time?
- Each stage can only work once the previous stage has produced activations for its microbatch. At the start, later stages wait for input; at the end, earlier stages run out of work first. That fill and drain period is the bubble. More microbatches can reduce the relative waste, but increase in-flight memory and often affect latency.
- Why is pipeline parallelism often better than tensor parallelism across nodes?
- Pipeline parallelism sends activations between neighbouring stages. Tensor parallelism needs frequent collectives inside layers, which are sensitive to network latency and bandwidth. Across slower interconnects, those repeated synchronisations can dominate. Pipeline parallelism trades them for fewer boundary transfers, provided stage compute is large enough to hide communication.
- Does pipeline parallelism help both prefill and decode equally?
- No. Prefill usually has large matrix work and can be split into useful microbatches, so the pipeline can be kept busier. Decode is more sequential because each next token depends on the previous sampled token. Continuous batching, prefix caching and speculative decoding can improve utilisation, but they do not remove that dependency.
Short definition: what is Pipeline Parallelism?