Module 06
The Fleet
One GPU becomes many, and many users become one bill. Sharding a model across devices, serving mixture-of-experts, routing requests so the cache still hits, and arriving at a defensible cost per million tokens.
12 lessons · 12 videos · 2h 7m
- 06.01
When One GPU Is Not Enough
Name the three separate reasons to shard: the weights do not fit, the KV cache does not fit, or the latency target is below what one device can reach. Recognise that they call for different strategies.
- 06.02
Tensor Parallelism
Split a matrix multiply across devices column-wise then row-wise, account for the all-reduce every layer pays, and explain why TP stops being worth it outside one NVLink domain.
- 06.03
Pipeline Parallelism
Split a model by layer across devices, describe the bubble that idle stages create, and say when PP is the right answer across nodes where TP's collectives are too expensive.
- 06.04
The Interconnect
Compare NVLink and NVSwitch against PCIe and InfiniBand by bandwidth and latency, and use the cost of an all-reduce as the real constraint on how far a model can be sharded.
- 06.05
Mixture of Experts
Explain why MoE changed the serving problem rather than just the training one: compute per token stays small while the memory that must be resident stays large, so the bottleneck moves.
- 06.06
Expert Parallelism
Place experts across devices, trace the all-to-all that routing requires, and explain why expert load imbalance, not raw FLOPs, is what limits a wide-EP deployment.
- 06.07
Routing Requests
Route across replicas by KV-cache locality and session affinity, and explain why plain round-robin destroys the prefix hit rate the KV cache module worked to build.
- 06.08
Multi-Tenancy and LoRA
Serve many fine-tunes from one base model by swapping low-rank adapters per request, and state what that buys over one deployment per tenant.
- 06.09
Autoscaling and Cold Starts
Choose between scale-to-zero and a warm pool by pricing the cold start honestly: loading tens of gigabytes of weights is the cold start, and no amount of container tuning removes it.
- 06.10
Choosing the Hardware
Compare H100, H200, B200 and rack-scale GB200 NVL72 on the axes that decide inference cost rather than on peak FLOPs: memory capacity, memory bandwidth, interconnect and native low-precision support.
- 06.11
Cost per Million Tokens
Build the unit-economics model from GPU-hour price, measured tokens per second and realistic utilisation, and produce separate input and output prices the way every provider quotes them.
- 06.12
Self-Host or API
Decide between self-hosting and a hosted API on measured numbers: break-even volume, latency requirements, data residency and the engineering time a fleet actually costs. Accept that the API often wins.