GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh2023ICLR 2023
Read it on arxiv.org(opens in a new tab)Why this one
Read this after LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. LLM.int8() shows that transformer inference has enough numerical slack to survive lower precision, GPTQ shows how far that slack goes when you treat quantization as a local reconstruction problem instead of a blunt rounding pass. The useful idea is not just "use 4 bits". It is that each quantized weight can be chosen while accounting for the damage it causes to the layer output, using a cheap approximation to second-order information. People often talk about GPTQ as a storage trick, but for an engineer the lesson is sharper: compression quality is tied to calibration data, kernel layout, and the order in which errors are absorbed. This is worth your evening because it explains why some 4-bit models feel real in production and others feel cursed, even when the headline bit width is identical.
What to take away
- GPTQ quantizes weights one layer at a time, correcting later weights for errors introduced by earlier quantized weights.
- The method uses calibration activations, so the data you choose affects how well the compressed model behaves.
- The paper separates model size reduction from actual speedup, which depends on kernels, packing, and hardware.
Reads with
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
shows the earlier 8-bit case that makes 4-bit post-training quantization plausible
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
attacks the same activation outlier problem by moving scale between activations and weights
- AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration
turns the calibration lesson into an activation-aware recipe for on-device 4-bit models