How is pipeline parallelism different from tensor parallelism?

Pipeline parallelism splits the model by layers, so each device owns a vertical slice of the network and sends activations onward. Tensor parallelism splits individual tensor operations, such as projections, across devices and then combines partial results with collectives. Pipeline parallelism communicates less frequently, but introduces stage idle time and sequential dependencies.

What is the pipeline bubble?

The pipeline bubble is the fraction of time when some stages have no useful work because the pipeline is being filled or drained. At the start, later stages wait for activations to arrive. At the end, earlier stages finish before later ones. More microbatches can shrink the bubble, but they add memory pressure and scheduling delay.

When is pipeline parallelism the right choice for inference?

It is a good candidate when the model or KV cache cannot fit on one device, or when tensor parallel collectives would cross slow inter-node links. It depends on batch shape, prefill versus decode mix, stage balance, activation size, and network bandwidth. If the model fits on one GPU, it is usually unnecessary overhead.

Pipeline Parallelism

Pipeline parallelism is a way to run one neural network across multiple devices by assigning different contiguous layers to different stages. Each stage computes its slice, passes activations to the next stage, and keeps its own weights and KV state, trading memory capacity and cheaper inter-node communication for pipeline idle time.

Pipeline parallelism becomes necessary when a model, its KV cache, and runtime workspace do not fit comfortably on one device, or when tensor parallelism would require collectives over links that are too slow. Instead of splitting every matrix operation across devices, it splits the model by depth. This is especially relevant across nodes, where sending activations between neighbouring stages can be cheaper than synchronising many partial results inside every transformer block.

Concretely, the layer stack is cut into ordered ranges. Stage one runs the early layers for a microbatch, sends the resulting activations to stage two, and so on until the final stage produces logits. To keep stages busy, a larger batch is divided into microbatches so different stages can work on different microbatches at the same time. The catch is the fill and drain period: early on, later stages wait; at the end, earlier stages wait. That idle region is the pipeline bubble.

The main trade-off is latency and scheduling complexity. More microbatches reduce the bubble, but increase in-flight activation memory and can delay the first visible result. Uneven stages make the slowest stage set the pace, so layer placement, embeddings, logits, MoE routing, and KV distribution matter. Pipeline parallelism is commonly misunderstood as simply adding more GPUs for speed. Often it is a capacity and communication strategy first, and a throughput win only when the pipeline stays full.

Engineers meet pipeline parallelism in multi-GPU and multi-node inference engines, usually combined with tensor parallelism, paged KV allocation, continuous batching, and request routing. It tends to fit large prefill workloads better than tiny decode steps, because prefill has enough compute and microbatching opportunity to hide transfers. Inside a node with fast interconnects, tensor parallelism may be preferable. Across slower node links, pipeline parallelism is often the cleaner compromise.

Go deeper

Inference EngineeringPipeline ParallelismSplit a model by layer across devices, describe the bubble that idle stages create, and say when PP is the right answer across nodes where TP's collectives are too expensive.Open the lesson

Common questions

How is pipeline parallelism different from tensor parallelism?: Pipeline parallelism splits the model by layers, so each device owns a vertical slice of the network and sends activations onward. Tensor parallelism splits individual tensor operations, such as projections, across devices and then combines partial results with collectives. Pipeline parallelism communicates less frequently, but introduces stage idle time and sequential dependencies.
What is the pipeline bubble?: The pipeline bubble is the fraction of time when some stages have no useful work because the pipeline is being filled or drained. At the start, later stages wait for activations to arrive. At the end, earlier stages finish before later ones. More microbatches can shrink the bubble, but they add memory pressure and scheduling delay.
When is pipeline parallelism the right choice for inference?: It is a good candidate when the model or KV cache cannot fit on one device, or when tensor parallel collectives would cross slow inter-node links. It depends on batch shape, prefill versus decode mix, stage balance, activation size, and network bandwidth. If the model fits on one GPU, it is usually unnecessary overhead.

Every term