06.04 · Concept
The Interconnect
Compare NVLink and NVSwitch against PCIe and InfiniBand by bandwidth and latency, and use the cost of an all-reduce as the real constraint on how far a model can be sharded.
Interconnect choice sets the practical boundary for model sharding: NVLink and NVSwitch make frequent intra-node collectives viable, while PCIe and InfiniBand often turn tensor-parallel decode into synchronisation overhead. The key test is not whether weights fit across GPUs, but whether each all-reduce still fits inside the token latency budget.
What this lesson answers
- NVLink versus PCIe for tensor parallel inference
- when does all-reduce limit model sharding
- why InfiniBand is slow for decode parallelism
Notes
The interconnect is the fabric that turns per-GPU inference into a distributed memory hierarchy, and its governing penalty is the collective time inserted into each decoder step. For tensor parallel inference, each layer typically performs one or two cross-GPU reductions after sharded matrix products; the ideal ring all-reduce cost is , where is tensor-parallel degree, is bytes reduced per GPU, is effective link bandwidth, and is per-hop/message latency.
Common questions
- Why is NVSwitch usually better than InfiniBand for tensor parallel inference?
- Tensor parallel inference performs collective communication inside the layer path, often during every decoder step. NVSwitch keeps those collectives within a high-bandwidth, low-latency GPU fabric. InfiniBand is built for node-to-node communication, so each collective pays extra latency through NICs, PCIe, RDMA paths and switches.
- Is bandwidth enough to compare GPU interconnects?
- No. Decode often moves relatively small activation buffers, so latency can dominate even when nominal bandwidth looks adequate. Prefill and larger batches make bandwidth more important, but tensor-parallel serving still has synchronisation on the critical path. A useful comparison includes message size, collective pattern, bandwidth and per-message latency.
- How do I know if a model is sharded too far?
- A strong warning sign is that all-reduce time becomes a visible part of inter-token latency. Adding more shards may reduce local matrix work, but if it pushes reductions over PCIe or InfiniBand, the system can become more serial, not more parallel. Aggregate memory capacity is not the real limit.
Short definition: what is Interconnect?