Fast Transformer Decoding: One Write-Head is All You Need
Noam Shazeer2019arXiv 2019
Read it on arxiv.org(opens in a new tab)Why this one
Read this before GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. Shazeer’s point is brutally practical: during autoregressive decoding, attention is often not compute bound, it is memory bandwidth bound, because every new token rereads a growing KV cache. Multi-query attention asks whether every query head really needs its own keys and values. Usually, no. Share one set of K and V across heads, keep many Q heads, and the cache shrinks by roughly the number of heads. People get this wrong when they treat attention variants as model quality trivia. This is an inference systems paper hiding inside a modeling paper. If you serve LLMs, it explains why architecture choices made during training determine batch size, latency, and cost years later.
What to take away
- Multi-query attention keeps many query heads but shares one key and value head across them.
- The main win is lower KV cache bandwidth during token-by-token decoding.
- The quality tradeoff is small enough that later production models made this idea standard.
Reads with
- Attention Is All You Need
defines the multi-head kv cache that decoding later trips over
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
generalizes mqa by sharing keys and values across groups instead of all heads
- Efficient Memory Management for Large Language Model Serving with PagedAttention
turns the same kv-cache bottleneck into a serving-time memory management problem