Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel, Noam Shazeer, Adam Roberts, et al.2020JMLR 2020
Read it on arxiv.org(opens in a new tab)Why this one
Read this after BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding and Sequence to Sequence Learning with Neural Networks. T5 is the paper that made a boring interface decision feel inevitable: cast every NLP job as text in, text out, then reuse one model, one loss, and one fine-tuning recipe. People often remember it as a benchmark sweep or a bigger Transformer, but the durable idea is the product surface for learning. Classification labels become tokens, translation is just another target string, and task design moves from custom heads into prompts and datasets. That is the road from supervised NLP systems to the prompt era later made obvious by Language Models are Few-Shot Learners. Worth your evening because it teaches you how much mileage comes from removing special cases, not inventing a new block diagram.
What to take away
- T5 turns classification, QA, summarization, and translation into the same supervised sequence task.
- The task prefix is an early, practical version of prompting, not decoration.
- Its ablations are useful when choosing objectives, data cleaning, model size, and fine-tuning setup.
Reads with
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
t5 generalizes bert style pretraining into a single text to text recipe
- Sequence to Sequence Learning with Neural Networks
t5 turns the encoder decoder translation setup into a universal nlp interface
- Language Models are Few-Shot Learners
the text to text interface becomes the prompt surface for few shot language models