Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al.2020NeurIPS 2020
Read it on arxiv.org(opens in a new tab)Why this one
Every RAG stack in production is a descendant of this paper, and most of them skip the part that made it interesting. The idea people remember is 'look things up before you answer'. The idea worth reading for is that the retriever and the generator are trained together, so the model learns which documents actually help rather than being handed whatever a similarity search returned. Read it and you stop treating retrieval as a preprocessing step you bolt on, and start treating the retriever as a component with its own failure modes, its own metrics, and its own responsibility for the answer. It also gives you the vocabulary - parametric versus non-parametric memory - that makes the rest of the retrieval literature readable.
What to take away
- Retrieval is part of the model, not a wrapper around it.
- A wrong answer has two possible causes now: the retriever missed, or the generator ignored what it was given. Measure them separately.
- Non-parametric memory is the cheapest way to update what a model knows without touching a weight.
Reads with
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
its dense retriever depends on pretrained bidirectional encoders
- Attention Is All You Need
the generator side is a transformer sequence model with cross attention
- ReAct: Synergizing Reasoning and Acting in Language Models
turns retrieval from a hidden memory lookup into an explicit reasoning action