Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann, Nick Ryder, et al.2020NeurIPS 2020
Read it on arxiv.org(opens in a new tab)Why this one
The useful idea here is not that GPT-3 was big. It is that a language model trained only to predict text can be steered at inference time by examples placed in its context, with no weight update and no task-specific head. People often treat few-shot prompting as a bag of prompt hacks, but the paper is really about a new interface to models: describe the task in the same medium the model was trained on, then let scale turn pattern completion into weak adaptation. Read it with a builder's eye. The examples are not magic words, they are temporary training data living inside the context window. That framing explains why ordering, format, label choice, and contamination matter, and why later work on instruction tuning, RAG, agents, and evals all had to take in-context learning seriously.
What to take away
- Few-shot examples act like task data inside the prompt, not like parameters learned by gradient descent.
- The same model can switch tasks by changing context, which makes evaluation depend heavily on prompt format.
- Scaling improved in-context learning enough to make prompting a real engineering interface.
Reads with
- Attention Is All You Need
the transformer architecture makes large promptable language models practical
- Scaling Laws for Neural Language Models
it explains why scale turns next-token prediction into broader task competence
- Training language models to follow instructions with human feedback
it turns the prompting interface into a model trained to follow user intent