How is an interconnect different from a network?

A network moves data between machines in general. An interconnect, in this context, is the performance-critical fabric used by accelerators to communicate during model execution. It includes local GPU fabrics such as NVLink and NVSwitch, host-attached paths such as PCIe, and cluster fabrics such as InfiniBand when collectives cross machine boundaries.

Why does all-reduce matter so much for inference?

Tensor parallel inference splits matrix operations across GPUs, then must combine partial results before the next dependent computation can continue. That combination is often an all-reduce. Because decoder layers run in sequence, collective time is inserted repeatedly on the critical path, so small delays per layer can dominate token latency.

Can software hide interconnect cost?

Sometimes, but not always. Fused collectives, better scheduling, and overlapping communication with GEMMs can hide part of the cost when computation is large enough. In small decode steps there may be little independent work to overlap, so the collective becomes exposed. The honest answer depends on batch shape, topology, kernels, and parallelisation strategy.

Is InfiniBand good enough for tensor parallelism?

It depends, but it is usually a poor fit for frequent decoder-layer reductions compared with NVLink and NVSwitch. InfiniBand can be excellent for connecting nodes, moving larger messages, pipeline boundaries, expert traffic, or disaggregated serving. It does not behave like an intra-node GPU fabric when every layer needs synchronised collectives.

Interconnect

An interconnect is the communication fabric that lets GPUs exchange activations, gradients, KV data, or collective results while running a model across more than one device. In inference, it is often the real limit on sharding: every saved local computation can add synchronisation over NVLink, NVSwitch, PCIe, or InfiniBand.

Large models are split because one GPU may not have enough memory or enough compute to serve the target workload. The trap is that sharding does not only divide work, it also creates messages. Tensor parallelism is especially sensitive because each decoder step depends on reductions between GPUs after sharded matrix operations. A model can fit across a fleet and still be slow because token generation waits on communication rather than arithmetic.

Concretely, the interconnect carries collective operations such as all-reduce, all-to-all, and point-to-point sends. In a ring all-reduce, each GPU passes chunks of a tensor around its neighbours, accumulating partial results until every GPU has the same reduced value. NVLink connects GPUs directly at high bandwidth and low latency; NVSwitch extends that into near all-to-all connectivity inside a node. Across nodes, InfiniBand usually carries traffic through NICs and switches, with more latency in the path.

The cost is not just bandwidth. Decode microbatches often move small activation tensors, so per-message latency and synchronisation dominate. Prefill can be more bandwidth-bound because messages are larger. This is commonly misunderstood: a faster network link does not automatically make cross-node tensor parallelism efficient. It depends on tensor-parallel degree, message size, layer count, collective implementation, topology, and whether compute is large enough to hide communication.

Engineers meet interconnect limits in traces from NCCL collectives, GPU profilers, and serving metrics such as inter-token latency. NVSwitch is commonly the practical boundary for aggressive tensor parallel inference, while InfiniBand is more often used for pipeline stages, disaggregated prefill and decode, expert routing, or replica-level traffic. PCIe-only servers can be especially deceptive: aggregate memory may look sufficient, but reductions through host bridges can turn parallelism into serial waiting.

Go deeper

Inference EngineeringThe InterconnectCompare NVLink and NVSwitch against PCIe and InfiniBand by bandwidth and latency, and use the cost of an all-reduce as the real constraint on how far a model can be sharded.Open the lesson

Common questions

How is an interconnect different from a network?: A network moves data between machines in general. An interconnect, in this context, is the performance-critical fabric used by accelerators to communicate during model execution. It includes local GPU fabrics such as NVLink and NVSwitch, host-attached paths such as PCIe, and cluster fabrics such as InfiniBand when collectives cross machine boundaries.
Why does all-reduce matter so much for inference?: Tensor parallel inference splits matrix operations across GPUs, then must combine partial results before the next dependent computation can continue. That combination is often an all-reduce. Because decoder layers run in sequence, collective time is inserted repeatedly on the critical path, so small delays per layer can dominate token latency.
Can software hide interconnect cost?: Sometimes, but not always. Fused collectives, better scheduling, and overlapping communication with GEMMs can hide part of the cost when computation is large enough. In small decode steps there may be little independent work to overlap, so the collective becomes exposed. The honest answer depends on batch shape, topology, kernels, and parallelisation strategy.
Is InfiniBand good enough for tensor parallelism?: It depends, but it is usually a poor fit for frequent decoder-layer reductions compared with NVLink and NVSwitch. InfiniBand can be excellent for connecting nodes, moving larger messages, pipeline boundaries, expert traffic, or disaggregated serving. It does not behave like an intra-node GPU fabric when every layer needs synchronised collectives.

Every term