Efficient Transformers: A Survey
Yi Tay, Mostafa Dehghani, Dara Bahri, Donald Metzler2022ACM Computing Surveys 2022
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Attention Is All You Need and before FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. The useful idea here is not one trick, but a taxonomy: most efficient Transformer papers buy speed or length by changing sparsity, low-rank structure, recurrence, memory, or kernel computation, and each bargain has a bill. People get this literature wrong by treating every quadratic-attention replacement as a drop-in improvement. Many wins live inside a benchmark shape, a sequence length, a hardware assumption, or a quality tolerance you may not share. This survey gives you the map you need before you rewrite an attention layer or believe a chart. It is worth an evening because inference work is full of plausible shortcuts, and this paper teaches you to ask what was actually saved: FLOPs, bytes moved, cache pressure, latency, or merely asymptotic embarrassment.
What to take away
- Classify an efficient attention paper by its mechanism before you compare its numbers.
- Check whether the win is in training, decoding, long-context prefill, or all three.
- Approximate attention is only useful if the task tolerates the approximation.
Reads with
- Attention Is All You Need
the baseline architecture whose quadratic attention all these variants are trying to escape
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
turns the survey's kernel efficiency bucket into the dominant exact-attention implementation
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
shows the recurrent state-space branch becoming a serious transformer alternative