03.05 · Concept
MHA, MQA, GQA
Compare multi-head, multi-query, and grouped-query attention. Explain that these are memory decisions --- not architecture choices --- driven by the need to shrink the KV cache. Understand how GQA shares key-value heads across query heads.
MHA, MQA and GQA mainly differ in how many key-value histories must be stored and read during autoregressive decode. MHA keeps separate K/V streams per query head, MQA shares one across all query heads, and GQA shares each K/V stream across a group of query heads to shrink the KV cache.
What this lesson answers
- MHA MQA GQA difference for KV cache
- how grouped query attention shares KV heads
- why GQA reduces inference memory bandwidth
Notes
Multi-head attention stores one key and one value vector per layer, token, and KV head, while query heads may be more numerous: with layers, context length , batch size , KV heads , head dimension , and element size bytes, the decode-time KV cache footprint is . In classic MHA, ; in MQA, ; in GQA, and each KV head is shared by query heads.
References
Common questions
- Is GQA a different transformer architecture or a serving optimisation?
- GQA is best understood as a memory layout decision for inference. The model can still have many query heads, but fewer distinct key-value streams are cached and read during decode. That reduces KV cache capacity and bandwidth pressure without simply removing query heads from the computation.
- How does grouped-query attention share key and value heads?
- In GQA, several query heads are assigned to the same key-value head. Each query head still produces its own attention scores, but it scores against a shared K/V history for its group. The cache stores only the smaller set of K/V streams, not repeated copies for every query head.
- When does MQA or GQA not improve latency much?
- The benefit is small when KV cache bandwidth or capacity is not the limiting factor. Short contexts, small batches, and prefill-heavy workloads are often dominated by matrix multiplications and attention score construction. In those cases, reducing stored K/V heads may save memory without noticeably improving end-to-end latency.