Distilling the Knowledge in a Neural Network
Geoffrey Hinton, Oriol Vinyals, Jeff Dean2015arXiv 2015
Read it on arxiv.org(opens in a new tab)Why this one
The useful idea here is not that a small model can copy a big model. It is that the big model's wrong answers contain signal. A label says “cat”; a softened teacher distribution says “cat, but fox and dog are the nearby mistakes, not toaster.” That ranking is often a better training target than the dataset itself. People still get distillation wrong by treating it as generic compression, then measuring only final accuracy. For an engineer, the real question is whether you can spend training compute once to buy cheaper latency, memory, and fleet cost forever. Read this after Scaling Laws for Neural Language Models if you want the practical counterweight: scaling tells you why the teacher is expensive, distillation tells you how to stop serving it everywhere. It also sets up later inference papers like LLM.int8() and GPTQ, which shrink numbers, while this shrinks behavior.
What to take away
- Soft targets teach similarity between classes, not just the winning label.
- Temperature is the knob that exposes dark knowledge in the teacher distribution.
- Distillation trades offline training cost for lower online serving cost.
Reads with
- Scaling Laws for Neural Language Models
scaling explains why the teacher is too expensive to serve everywhere
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
quantization shrinks arithmetic while distillation shrinks behavior
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
post-training compression attacks the same serving cost from the weight side