SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
Guangxuan Xiao, Ji Lin, Mickael Seznec, et al.2023ICML 2023
Read it on arxiv.org(opens in a new tab)Why this one
Read this after LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, because it attacks the same wall from the opposite side. The trap in LLM quantization is thinking the weights are the scary part. For many transformer blocks, the real problem is a few huge activation channels that wreck the scale for everyone else. The useful move here is not a new quantizer, but a reparameterization: shift the outlier burden from activations into weights, where offline calibration and per-channel scaling make it easier to absorb. People often cite this as just another compression paper, but for an inference engineer it is really about making INT8 matmuls line up with hardware instead of fighting the model. Read GPTQ and AWQ afterward for the weight-only branch of the family tree.
What to take away
- Activation outliers, not weight ranges, are the main reason naive W8A8 breaks on LLMs.
- SmoothQuant rescales channels so activations become easier to quantize while weights take the extra range.
- The method is post-training, so it belongs in a serving pipeline rather than a pretraining recipe.
Reads with
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
sets up activation outliers as the core obstacle smoothquant relocates
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
takes the other branch by quantizing weights while leaving activations higher precision
- AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration
continues the weight-only line using activation statistics to protect important weights