AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration
Ji Lin, Jiaming Tang, Haotian Tang, et al.2024MLSys 2024
Read it on arxiv.org(opens in a new tab)Why this one
Read this after GPTQ and SmoothQuant. GPTQ treats quantization like a careful repair job on weights, while SmoothQuant moves difficulty between activations and weights. The useful idea here is simpler and very engineer-friendly: your calibration activations are a map of which weight channels must survive low precision. People often hear AWQ as “keep 1 percent in higher precision,” but that is exactly the trap the paper avoids. Mixed precision is awkward on real hardware, so AWQ uses scaling to protect important channels while still running uniform low-bit weight kernels. That is why it matters for builders, not just benchmark tables. If you are trying to put a model on a laptop, phone, robot, or cheap GPU, this paper teaches the right question to ask before writing kernels: which errors will the model actually feel at runtime?
What to take away
- Activation statistics identify salient weight channels better than weight magnitude alone.
- AWQ protects important channels with scaling, avoiding mixed-precision kernels.
- The method needs no backpropagation, so calibration is lightweight and less domain-fragile.
Reads with
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
sets up weight-only post-training quantization as an error repair problem
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
shows the activation-weight tradeoff that awq exploits without mixed precision
- The case for 4-bit precision: k-bit Inference Scaling Laws
explains why pushing below 8-bit matters for real serving cost