Activation Quantisation
Activation quantisation is the conversion of a model’s intermediate tensors, not just its stored weights, into low-bit numeric formats during inference. It lets linear operations use integer kernels and move fewer bytes, but transformer activations contain outlier channels that make naive scaling lose information across most of the tensor.
Weight quantisation is comparatively tidy because weights are fixed before serving and can be analysed, grouped, and assigned scales offline. Activations are produced for each token as the model runs, so their scales must be chosen cheaply and repeatedly. In transformers, a few hidden channels often have much larger magnitudes than the rest. If one scale must cover the whole activation tensor, those rare large values determine the range and smaller values get rounded coarsely or to zero.
In the common integer path, an activation tensor is divided by a scale, rounded, clipped into the integer range, multiplied with similarly quantised weights, then interpreted using the activation and weight scales. SmoothQuant changes the geometry before quantisation. For a linear layer, it rewrites the product by dividing selected activation channels by a positive scale and multiplying the corresponding weight columns by the same scale, so the real-valued layer output is unchanged before rounding.
The trade is not free. SmoothQuant moves range from activations into weights because weight scales are static and can often be per channel, where the extra range is less damaging. It works best when outliers are persistent by channel and visible during calibration. If outliers are token-specific, spread across many channels, or the target precision is very low, the shifted weight error can cancel the activation benefit.
Engineers usually meet activation quantisation in W8A8-style inference engines, export pipelines, and post-training quantisation flows. Attention softmax, normalisation, RoPE, and residual paths are often still kept in FP16 or BF16 because forcing every intermediate edge into integers can hurt accuracy or add conversion overhead. It is commonly confused with weight-only quantisation, but the hard part is scaling live tensors, not compressing stored matrices.
Common questions
- Why is activation quantisation harder than weight quantisation?
- Weights are known in advance, so their distributions can be inspected and quantised with static per-channel or grouped scales. Activations depend on the prompt, token position, and layer state. A small number of activation channels can dominate the observed range, making a single scale waste most integer levels on values that rarely occur.
- What does SmoothQuant actually do?
- SmoothQuant uses the fact that a linear layer can be rescaled without changing its real-valued output. It divides troublesome activation channels by chosen scale factors and multiplies the matching weight columns by those same factors. The activation tensor becomes easier to quantise, while the weights absorb the extra range where static scaling is cheaper.
- Does activation quantisation always make inference faster?
- No. It depends on the hardware, batch shape, kernel fusion, memory bandwidth, and which operators remain in higher precision. Integer GEMMs and reduced activation traffic can help, but extra quantise, dequantise, scale loading, and unfused residual or normalisation work can erase the gain in some serving regimes.
- Is KV-cache quantisation the same thing as activation quantisation?
- It is related but not identical. KV cache tensors are activation-derived state stored across decoding steps, so quantising them also saves memory traffic and capacity. However, the error pattern and serving impact differ from quantising the inputs to linear layers, and many systems treat KV-cache quantisation as a separate feature.