06.10 · Concept
Choosing the Hardware
Compare H100, H200, B200 and rack-scale GB200 NVL72 on the axes that decide inference cost rather than on peak FLOPs: memory capacity, memory bandwidth, interconnect and native low-precision support.
Inference hardware cost is usually set by memory capacity, HBM bandwidth, interconnect, and usable low-precision formats, not headline FLOPs. H100, H200, B200, and GB200 NVL72 make different trade-offs around weight streaming, KV cache residency, tensor parallel communication, and whether FP8 or FP4 can be used without hurting model quality.
What this lesson answers
- H100 vs H200 for inference serving cost
- when does B200 improve LLM inference economics
- is GB200 NVL72 worth it for inference
Notes
For inference hardware selection, the governing mechanism is not peak FLOPs but the bottleneck term in the decode step: . For a single autoregressive decode stream, because each generated token rereads the model weights, while the KV footprint is .
References
Common questions
- Why are peak FLOPs a poor way to choose inference GPUs?
- Autoregressive decode often waits on model weights being read from HBM, KV cache movement, or cross-GPU communication. If those dominate, extra tensor throughput does little for latency or cost per token. FLOPs matter once batching, quantisation, and kernels move the workload into a compute-bound regime.
- When is H200 a better inference choice than H100?
- H200 is attractive when capacity and bandwidth reduce sharding, KV evictions, or context limits. It can let a replica admit larger batches or longer sessions with less operational friction. If traffic is latency-capped at small batch and the model already fits cleanly, much of the extra memory may not pay for itself.
- What workloads justify GB200 NVL72 instead of smaller GPU nodes?
- GB200 NVL72 fits scale-up inference: very large models, heavy tensor parallelism, MoE serving, disaggregated prefill and decode, or many long-context sessions where KV placement and cross-GPU bandwidth matter. For independent smaller replicas with moderate context, its rack-scale fabric and reserved power can be underused.