An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al.2021ICLR 2021
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Attention Is All You Need and ImageNet Classification with Deep Convolutional Neural Networks. The useful shock here is not that images can be split into patches, it is that the old vision prior of locality and translation baked into convolutions was a convenience, not a law. ViT treats a picture as a sequence and lets scale plus pretraining learn the structure. People often retell this as "CNNs are dead", which misses the engineering lesson. On small data, the missing inductive bias hurts. With enough data and a clean transfer setup, the simpler general architecture wins because it is easier to scale and reuse. If you are building models, this paper is worth an evening because it teaches when architecture should encode domain knowledge, and when your real product is the pretraining pipeline.
What to take away
- A fixed size image patch becomes the vision equivalent of a token.
- ViT works best when large scale pretraining replaces convolutional bias.
- The class token and positional embeddings are enough to make vanilla Transformer machinery usable for images.
Reads with
- Attention Is All You Need
the transformer block vit ports from text to image patches starts here
- ImageNet Classification with Deep Convolutional Neural Networks
vit is the large scale answer to the convolutional prior alexnet made dominant
- Learning Transferable Visual Models From Natural Language Supervision
clip shows why a scalable transformer vision backbone matters beyond classification