05.04 · Concept
Activation Quantisation
Explain why quantising activations as well as weights is harder than weights alone: a few outlier channels carry a disproportionate range, and how SmoothQuant migrates that difficulty into the weights where it is cheap.
Activation quantisation is difficult because transformer activations can have a few channels with much larger ranges than the rest, forcing coarse shared scales that damage ordinary values. SmoothQuant preserves the linear result by rescaling activations down and weights up, moving the awkward range into statically quantised weights where per-channel scales are cheap.
What this lesson answers
- why is activation quantisation harder than weight quantisation
- how does SmoothQuant handle activation outlier channels
- when does W8A8 quantisation hurt inference quality
Notes
Activation quantisation replaces high-precision intermediate tensors in the transformer, especially the inputs to linear layers, with low-bit integers so GEMMs can run as rather than in FP16/BF16. For per-tensor INT8, with ; weights use analogous scales, often per output channel.
References
Common questions
- Why are activations harder to quantise than weights?
- Weights are fixed, so their scales can be calibrated offline and often chosen per output channel. Activations arrive at runtime, vary by token and layer, and may contain a few hidden channels with unusually large magnitudes. A shared activation scale then follows those outliers, leaving common values represented with very few useful integer levels.
- What does SmoothQuant change in the model?
- SmoothQuant applies an exactly cancelling rescale around linear layers. It divides selected activation channels by a positive scale and multiplies the matching weight columns by the same scale. The real-valued matrix product is unchanged, but the activation tensor becomes easier to quantise while the extra range lands in weights that can use static per-channel quantisation.
- Is activation quantisation always faster in production?
- No. It helps when integer GEMMs and reduced activation traffic dominate the path. It can regress when the workload is already compute-bound, when scale handling and conversions are not fused, or when sensitive operations such as attention softmax, residual paths, RoPE, and normalisation would lose too much accuracy if forced into low precision.
Short definition: what is Activation Quantisation?