Number Formats

Notes

A reduced-precision number format is a contract for encoding a tensor value $x$ into fewer transported bits, usually as $x\approx s\cdot q$ where $q$ is a low-bit floating or integer code and $s$ is an implicit, per-tensor, per-channel, or per-block scale. BF16 keeps FP32’s 8-bit exponent and sign but cuts the mantissa to 7 bits, so it costs 2 bytes and preserves dynamic range; FP8 variants spend 1 byte, typically E4M3 for activations or E5M2 for gradients/weights depending on range; INT8 spends 1 byte on a signed integer code and moves the exponent entirely into $s$; INT4 spends half a byte and relies even more heavily on scale granularity. Microscaling formats make that movement explicit: MXFP4 stores 4-bit element codes plus a shared block scale, commonly one 8-bit scale per 32 values in OCP MX formats, so the effective cost is $4+8/32=4.25$ bits/value before packing overhead. NVIDIA’s NVFP4, introduced with Blackwell Tensor Cores, similarly uses 4-bit floating values with fine-grained scaling, but is designed around NVIDIA’s native FP4 execution path rather than just storage compression.

The exponent bits are the key mental model. BF16 has the same exponent budget as FP32, so overflow behavior in transformer inference is forgiving even when logits, residuals, and attention scores vary across prompts; the price is that each multiply operand is still 16 bits. FP8 still has per-value exponent bits, but fewer of them: E4M3 gives more precision near 1 and less range, while E5M2 gives BF16-like range pressure relief with only two explicit mantissa bits. INT8 and INT4 have no per-value exponent; dequantization is $x_i\approx s_g(z_i-z_0)$ for a group $g$, so all values in that group share range and outliers consume the available codes. MXFP4/NVFP4 split the difference: the 4-bit payload may look float-like, but the usable dynamic range mostly comes from the shared microscale, so “where the exponent went” is partly into a block metadata stream and partly into a tiny local exponent field. This is why FP8 can often be activation-friendly, while INT4 is mostly a weight-only inference format unless the kernel, calibration, and model have been built around it.

On current hardware, native execution determines whether fewer bytes also mean fewer steps. NVIDIA H100 Tensor Cores natively accelerate BF16, FP16, TF32, FP8 E4M3/E5M2, INT8, and INT4 matrix instructions, but not NVIDIA’s Blackwell NVFP4 Tensor Core path; H100 can store FP4-like weights but cannot run NVFP4 as the intended native compute format. Blackwell B200/GB200 adds native FP4/NVFP4 support and very high FP8 throughput, which is why NVIDIA’s TensorRT-LLM releases emphasize FP8 on Hopper and FP4/NVFP4 on Blackwell. AMD MI300-class systems have strong BF16/FP16 and FP8 support through ROCm, but the exact FP8 variants and software maturity differ from CUDA. vLLM’s PagedAttention paper is not a number-format paper, but vLLM now exposes AWQ, GPTQ, FP8 KV cache, and bitsandbytes-style paths; SGLang similarly composes radix-cache serving with FP8 and weight quantization backends; llm-d and NVIDIA Dynamo treat precision as one scheduling dimension alongside disaggregated prefill/decode. FlashAttention is relevant because it keeps attention IO-aware; if the surrounding QKV and KV-cache tensors are FP8, the attention kernel must preserve the same layout and scaling discipline or the bytes saved disappear in conversions.

For a concrete bandwidth calculation, take an 80GB H100 with about $3.35$ TB/s HBM bandwidth and a 70B-parameter decoder model. If weights are streamed once per generated token in a memory-bound decode step, FP16 or BF16 weights cost $70\times 10^9\cdot 2=140$ GB per token just for weight reads, giving an optimistic ceiling of $3.35\times 10^{12}/140\times 10^9\approx 23.9$ tokens/s on one GPU before KV reads, activations, and collectives. INT8 weight-only cuts that to $70$ GB, so the roofline ceiling doubles to $47.9$ tokens/s if dequantization is fused into GEMM and tensor cores stay fed. INT4 nominally reads $35$ GB, so the ceiling is $95.7$ tokens/s; with MXFP4 at $4.25$ bits/value the model stream is $70\times 10^9\cdot 4.25/8=37.2$ GB and the ceiling is $90.1$ tokens/s. That arithmetic explains why TensorRT-LLM, vLLM, and SGLang invest in fused quantized GEMMs: an unfused path that reads INT4, expands to BF16 in HBM, then multiplies has paid both the 4-bit and 16-bit traffic and loses the point.

KV cache precision changes a different term. With grouped-query attention for a 70B-like model using 8 KV heads, head dimension 128, and 80 layers, each token stores keys and values of size $2\cdot 80\cdot 8\cdot 128=163,840$ elements. BF16 KV costs $327,680$ bytes/token, so a 32k-token context is about $10.5$ GB per sequence; FP8 KV costs $5.24$ GB, plus scale metadata if used. PagedAttention from the vLLM paper attacks fragmentation and paging of this cache, not the numerical format, but FP8 KV and paged allocation multiply: fewer bytes per block means more resident sequences and fewer evictions. FlashAttention reduces attention memory traffic for training and prefill by tiling SRAM, but during decode every new token still probes the existing KV history; FP8 KV directly lowers that stream. TensorRT-LLM’s FP8 KV cache, SGLang’s serving engine, NVIDIA Dynamo’s disaggregated serving stack, and llm-d’s Kubernetes-native inference design all exploit this same byte accounting when scheduling prefill-heavy versus decode-heavy traffic.

These formats stop helping when the saved HBM bytes are no longer the limiter, or when scales, conversions, and accuracy repairs dominate the path. Small batches and short contexts can become launch-latency or tensor-core-underfilled rather than bandwidth-bound; then INT4 may add dequant instructions without raising tokens/s. Prefill with large matrix multiplies can be compute-bound on H100, so FP8 often helps because it maps to native Tensor Cores, while weight-only INT4 may not if the kernel expands operands before MMA or uses slow groupwise scaling. Numerically, INT4 breaks on layers with high outlier channels, router logits, embedding/lm-head tying, and long-context attention unless group sizes shrink; but shrinking group size increases scale overhead, e.g. one FP16 scale per 16 INT4 weights adds $16/16=1$ extra bit/value, turning 4-bit into 5-bit before zero-points. Speculative decoders such as EAGLE, from the EAGLE paper on feature-level extrapolation for fast autoregressive generation, are especially sensitive to distribution drift: quantizing the draft model too aggressively can lower acceptance rate enough that fewer bytes per draft step produce more total verifier work.