Robust Speech Recognition via Large-Scale Weak Supervision
Alec Radford, Jong Wook Kim, Tao Xu, et al.2023ICML 2023
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Scaling Laws for Neural Language Models and before The Llama 3 Herd of Models. Whisper is the cleanest case study I know for a lesson engineers keep relearning: enough broad, ugly data can beat a smaller pile of carefully polished labels, especially when the deployment world is ugly too. The mistake is to read it as “just scale wins.” The useful idea is narrower: the training mixture matches the kind of variation users actually bring, accents, noise, languages, formats, timestamps, translation, and partial transcripts. That makes the model less like a benchmark specialist and more like a reusable system component. If you are building speech features, read it for the data design, not the architecture trivia. It will change what you log, what you filter, and what you stop overfitting to in evals.
What to take away
- Weak labels work when the dataset is huge, diverse, and close to real deployment noise.
- Multitask speech training makes transcription, translation, and language ID reinforce one shared model.
- Zero-shot evaluation is the point: robustness is measured by avoiding per-benchmark tuning.
Reads with
- Scaling Laws for Neural Language Models
gives the scaling frame behind why weak data becomes useful at enough volume
- Learning Transferable Visual Models From Natural Language Supervision
the same bet on messy web supervision, applied to vision instead of speech
- The Llama 3 Herd of Models
carries the lesson forward into general foundation model data mixture design