Learning Transferable Visual Models From Natural Language Supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, et al.2021ICML 2021
Read it on arxiv.org(opens in a new tab)Why this one
The useful idea here is not that images can have captions. It is that you can train two encoders to agree on the same point in vector space, then reuse that space for search, classification, filtering and retrieval without building a separate model for each label set. People often remember CLIP as a zero shot demo, but the more durable lesson is the data interface: natural language becomes the supervision layer. That changes how you design systems. Instead of asking for a fixed taxonomy, you ask whether your text queries, image encoder and similarity metric share enough structure to make ranking work. If you are building multimodal search or a RAG pipeline over screenshots, diagrams or product photos, this paper gives you the mental model behind the embedding API you are probably already calling.
What to take away
- CLIP trains image and text encoders with a contrastive loss over matched and mismatched pairs.
- Classification is turned into retrieval by comparing an image embedding with prompt embeddings for candidate labels.
- The paper shows why dataset scale and noisy natural language supervision can beat carefully curated fixed labels.
Reads with
- ImageNet Classification with Deep Convolutional Neural Networks
the fixed-label vision regime that clip replaces with language supervision
- Efficient Estimation of Word Representations in Vector Space
the earlier proof that useful meaning can live in reusable vector geometry
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
clip's shared embedding space becomes the retrieval layer for multimodal rag