Attention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, et al.2017NeurIPS 2017
Read it on arxiv.org(opens in a new tab)Why this one
The paper that took recurrence out of sequence modelling, and its argument is narrower and more mechanical than its reputation suggests. If you can compute how much every position should attend to every other position in a single matrix multiply, you no longer have to walk the sequence one step at a time - and hardware rewards that enormously. Almost everything people mean by 'the transformer' is in section three: scaled dot-product attention, several heads reading different subspaces, positional encodings added back because a set has no order, and a residual-plus-layer-norm sandwich that keeps a deep stack trainable. Read it once you can derive a dot product and a softmax by hand, and read it again if 'query, key, value' still sounds like naming rather than mechanism. The /labs attention heatmap exists to make that second reading land.
What to take away
- The scale factor of one over root d is not cosmetic - without it the softmax saturates and gradients vanish.
- Heads are the model looking at several relationships at once; a single head is genuinely weaker.
- Positional information is bolted on, and every improvement since (RoPE, ALiBi) is a better bolt.
Reads with
- Neural Machine Translation by Jointly Learning to Align and Translate
the alignment mechanism that transformers turn into the whole architecture
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
shows self-attention becoming a bidirectional pretraining substrate
- Language Models are Few-Shot Learners
scales the decoder-only half into the modern language model template