The case for 4-bit precision: k-bit Inference Scaling Laws
Tim Dettmers, Luke Zettlemoyer2023ICML 2023
Read it on arxiv.org(opens in a new tab)Why this one
If you serve models, this is the paper to read when the argument turns from taste to budget. The useful idea is not merely that 4-bit works, but that model size and precision are competing uses of the same memory envelope. A smaller clean model can lose to a larger rougher one, and the crossover is measurable. People often treat quantization as an implementation trick applied after model choice, then ask why quality moved. This paper makes precision part of model selection. Read this after LLM.int8(), GPTQ, SmoothQuant, and AWQ: those tell you how to quantize, this tells you what question to ask before you do. It is worth your evening because it gives you a mental table for the common production choice: spend bytes on more parameters, more precision, or more batch.
What to take away
- For a fixed bit budget, a larger 4-bit model often beats a smaller 8-bit model on zero-shot accuracy.
- Block size matters because smaller independently quantized blocks reduce the damage from outlier weights.
- Quantization method improvements help less than choosing the right precision and model scale in the first place.
Reads with
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
the baseline case for preserving outlier channels before pushing below 8 bit
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
turns the memory tradeoff into a practical post-training quantization method
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
shows the activation and weight rescaling trick behind many deployable low-bit models