05.03 · Walkthrough
Weight-Only Quantisation
Explain why INT4 weight-only quantisation is the natural first move for decode, since the weights are the bytes being read, and how AWQ and GPTQ decide what to round.
INT4 weight-only quantisation cuts decode latency by shrinking the model weights repeatedly streamed from memory, while leaving runtime activations mostly in higher precision. AWQ uses calibration activations to protect important channels; GPTQ uses a second-order layer objective to choose rounding and compensate remaining weights.
What this lesson answers
- why weight only quantisation helps decode
- how AWQ chooses what weights to protect
- how GPTQ rounding differs from nearest rounding
Notes
Weight-only quantisation stores the transformer weight matrices in low precision, typically INT8 or INT4, while leaving activations, residual stream, attention scores, KV cache, and usually accumulators in FP16/BF16. For a linear layer , inference stores for each quantisation group , with and for INT4; the kernel dequantises tiles of on the fly and performs tensor-core or mixed integer/F16 matmul.
References
Common questions
- Why is weight-only quantisation especially useful for decode?
- During autoregressive decode, each generated token causes the model to reread large weight matrices while doing relatively little work per layer. Storing those weights in INT4 directly reduces the dominant memory traffic. Activations, residuals and accumulators can stay in FP16 or BF16, so the change targets bandwidth without fully quantising the execution.
- What does AWQ use calibration activations for?
- AWQ looks at which activation channels matter most for a layer’s output, then rescales or protects the corresponding weights before quantising them. The deployed model still stores weights in low precision, but the rounding decision is informed by real activation statistics rather than treating every weight entry as equally important.
- How is GPTQ different from simple round-to-nearest quantisation?
- GPTQ quantises weights using a layerwise error model based on calibration inputs, not just the distance between each weight and its nearest quantised value. As it commits columns, it adjusts the remaining unquantised weights to compensate for the induced error, so rounding decisions reflect sensitivity of the layer’s computation.
Short definition: what is Weight-Only Quantisation?