LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer2022NeurIPS 2022
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Mixed Precision Training and before GPTQ, SmoothQuant, or AWQ. The useful idea here is that quantization failure in large transformers is not evenly spread noise, it is a small number of feature dimensions with huge activations that carry real work. Treat those outliers as structure, not as annoying measurement error, and int8 becomes a systems tool instead of a benchmark trick. People often remember this paper as “halve memory with no quality loss,” but the deeper lesson is diagnostic: when a compression method breaks at scale, inspect the distribution before inventing a new model. For an engineer, this is worth an evening because it explains why serving LLMs is often blocked by memory bandwidth and placement, not just FLOPs, and why the winning implementation is a mixed path that keeps the common case cheap while refusing to quantize the few values that matter.
What to take away
- Naive int8 fails because rare activation outliers dominate important transformer computations.
- Vector-wise quantization fixes most matrix multiplies by scaling per inner product, not globally.
- The outlier dimensions stay in fp16, so almost all work is int8 without sacrificing quality.
Reads with
- Mixed Precision Training
sets up the mixed numeric paths that llm int8 pushes into inference
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
turns the same memory pressure into post-training weight-only quantization
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
attacks the activation outlier problem by moving scale from activations into weights