A Mathematical Theory of Communication
Claude E. Shannon1948Bell System Technical Journal
Read it on doi.org(opens in a new tab)Why this one
Read this before Training language models to follow instructions with human feedback, BERT, or any paper where cross-entropy appears as a line of code. Shannon gives you the missing interpretation: information is not a vibe, it is the cost of resolving uncertainty under a distribution. The common mistake is to treat entropy as a metaphor for surprise, or cross-entropy as just the default classification loss. Here it becomes an engineering quantity. If your model assigns low probability to the thing that happens, you pay extra bits. If your codebook matches the source, you approach the limit. That idea runs straight through compression, token prediction, calibration, retrieval ranking, and why log loss is so hard to cheat. This is an old paper, but not a museum piece. It teaches you to see learning as modeling a source well enough that waste becomes measurable.
What to take away
- Entropy is the lower bound on average code length for messages drawn from a known distribution.
- Cross-entropy measures the extra bits you spend when your assumed distribution differs from the real source.
- The noisy channel framing separates source, encoder, channel, decoder, and destination, a design pattern still used in ML systems.
Reads with
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
masked language modeling turns shannon's coding cost into the training objective
- Scaling Laws for Neural Language Models
its loss curves measure wasted nats per token as models and data scale
- Training language models to follow instructions with human feedback
preference tuning still rests on log probabilities as calibrated information costs