02.08 · Concept
Precision
Compare FP32, FP16, BF16, INT8, and FP4. Understand the memory bandwidth savings and the accuracy tradeoffs. Read a dtype annotation on a model checkpoint and know what it implies for throughput.
Precision controls how many bytes inference must move for weights, activations, KV cache and accumulators. BF16 often gives FP16-like speed with safer range, while INT8 and FP4 cut memory traffic further at rising accuracy risk. A checkpoint dtype hints at capacity, bandwidth pressure, kernel choice and likely throughput.
What this lesson answers
- FP16 vs BF16 for transformer inference
- what does INT8 checkpoint dtype imply
- when does FP4 quantisation hurt accuracy
Notes
Precision is the choice of numeric format used to store weights, activations, KV cache, and intermediate accumulators; its first-order inference effect is set by bytes moved, not by the name of the dtype. For a bandwidth-bound decode step, the hardware floor is approximately and , where includes model weights and any KV traffic touched per generated token. FP32 stores 32 bits per scalar, FP16 and BF16 store 16, INT8 stores 8, and FP4 stores 4 before scale metadata.
Common questions
- Why is BF16 often preferred over FP16 for LLM inference?
- BF16 keeps the broad exponent range associated with FP32 while using the same storage width as FP16. That makes it less prone to overflow in transformer activations and reductions. On suitable GPUs, BF16 and FP16 usually have similar tensor-core throughput, so the practical difference is often numerical safety rather than raw bandwidth.
- Does INT8 always make inference twice as fast as BF16?
- No. INT8 reduces the raw bytes for stored values, which helps when decoding is limited by memory bandwidth. The gain can shrink when dequantisation, kernel scheduling, attention traffic, KV cache movement, batching, or application overhead dominate. It can also lose quality if activations or outlier channels are quantised too coarsely.
- What should I infer from a model checkpoint dtype annotation?
- A dtype annotation tells you how the checkpoint expects weights, and sometimes activations, to be stored or loaded. BF16 usually indicates a conventional high-quality serving path. INT8 may mean weight-only, activation, or KV quantisation, which have different risks. FP4 implies stronger compression but greater dependence on scaling and calibration.
Short definition: what is Precision?