Is BF16 always better than FP16?

No. BF16 is often safer because it has a much wider exponent range, so it is less prone to overflow in transformer activations. FP16 has more fraction detail within its smaller range. On modern GPU tensor cores their headline compute rates may be similar, so the difference is usually numerical robustness rather than raw bandwidth.

Why can INT8 or FP4 improve throughput?

They reduce the number of bytes read for weights or KV cache. If decode is bandwidth-bound, moving fewer bytes can directly improve tokens per second or fit more state in memory. The gain can disappear when prefill compute, attention traffic, launch overhead, dequantisation, or scheduling dominates instead of weight reads.

Does a quantised checkpoint mean the whole model runs in that precision?

Not necessarily. Weight-only quantisation may store weights as INT8 or FP4 while keeping activations, reductions, logits, and accumulators in FP16, BF16, or FP32. KV cache may also use a separate dtype. Always check what is quantised, the scale granularity, and whether the runtime has kernels for that exact format.

Why is FP4 more risky than INT8?

FP4 has far fewer representable levels, so scale choice becomes much more important. With poor group sizes or calibration, small but meaningful differences in weights or activations can be lost. It can work well with strong recipes such as quantisation-aware training, distillation, or careful post-training quantisation, but it is less forgiving.

Precision

Precision is the numeric format used to represent model weights, activations, KV cache entries, and accumulators during inference. It determines how many bytes must be read or written, what value range can be represented, and how much rounding or quantisation error the model must tolerate.

Precision matters because many inference workloads are limited by memory movement rather than arithmetic. During decode, the GPU repeatedly reads weights and touches KV cache for each generated token. Smaller formats reduce the bytes moved, so they can raise throughput or allow a larger model, batch, or context to fit. The dtype printed in a checkpoint is therefore not just metadata: it is a clue about bandwidth pressure, memory capacity, and kernel choice.

Floating-point formats trade range against detail. FP32 is wide and safe but expensive to move. FP16 and BF16 both use half the storage, but BF16 keeps FP32-like exponent range and gives up fraction bits, which is why it is often safer for transformers than FP16. INT8 and FP4 are quantised: stored integers approximate real values using a scale, sometimes with a zero-point, applied per tensor, channel, group, or block.

The cost is not only accuracy. Quantised weights usually need dequantisation before or during matrix multiplication, and scale metadata consumes space. Coarse scales can clip outlier channels; very small formats may blur residual-stream differences that matter for exact tokens, code, or structured output. The honest answer to “is lower precision faster?” is: it depends on whether HBM traffic is the bottleneck and whether the kernels fuse the conversion efficiently.

Engineers meet precision in checkpoint annotations such as BF16, FP16, INT8, FP8, or FP4, and in serving options for vLLM, SGLang, TensorRT-LLM, and similar runtimes. A BF16 model usually implies weight and activation storage suited to BF16 execution. An INT8 label is ambiguous: it may mean weight-only quantisation, activation quantisation, or KV cache compression, which have different performance and quality implications.

Go deeper

Inference EngineeringPrecisionCompare FP32, FP16, BF16, INT8, and FP4. Understand the memory bandwidth savings and the accuracy tradeoffs. Read a dtype annotation on a model checkpoint and know what it implies for throughput.Open the lesson

Common questions

Is BF16 always better than FP16?: No. BF16 is often safer because it has a much wider exponent range, so it is less prone to overflow in transformer activations. FP16 has more fraction detail within its smaller range. On modern GPU tensor cores their headline compute rates may be similar, so the difference is usually numerical robustness rather than raw bandwidth.
Why can INT8 or FP4 improve throughput?: They reduce the number of bytes read for weights or KV cache. If decode is bandwidth-bound, moving fewer bytes can directly improve tokens per second or fit more state in memory. The gain can disappear when prefill compute, attention traffic, launch overhead, dequantisation, or scheduling dominates instead of weight reads.
Does a quantised checkpoint mean the whole model runs in that precision?: Not necessarily. Weight-only quantisation may store weights as INT8 or FP4 while keeping activations, reductions, logits, and accumulators in FP16, BF16, or FP32. KV cache may also use a separate dtype. Always check what is quantised, the scale granularity, and whether the runtime has kernels for that exact format.
Why is FP4 more risky than INT8?: FP4 has far fewer representable levels, so scale choice becomes much more important. With poor group sizes or calibration, small but meaningful differences in weights or activations can be lost. It can work well with strong recipes such as quantisation-aware training, distillation, or careful post-training quantisation, but it is less forgiving.

Every term