Maximum Likelihood from Incomplete Data via the EM Algorithm
A. P. Dempster, N. M. Laird, D. B. Rubin1977Journal of the Royal Statistical Society B
Read it on doi.org(opens in a new tab)Why this one
Read this after Maximum Likelihood from Incomplete Data via the EM Algorithm only if you want the clean version of a pattern you have already used under other names: guess the missing structure, fit the model as if that guess were data, repeat. The trap is to treat EM as a clustering algorithm because k-means and Gaussian mixtures make it feel concrete. It is really a way to turn an ugly likelihood with hidden variables into a sequence of easier likelihood problems, with a monotonic improvement guarantee. That makes it useful far outside textbook mixtures: missing labels, alignments, topic models, old speech systems, and the intuition behind later variational methods. Read this before Auto-Encoding Variational Bayes, because VAEs are easier to understand once you have seen the older bargain: when exact inference is hard, optimize a surrogate you can actually compute.
What to take away
- EM alternates between estimating hidden structure and maximizing parameters under that estimate.
- The proof matters because each iteration cannot decrease the observed-data likelihood.
- K-means is best read as a hard-assignment special case, not the center of the idea.
Reads with
- Auto-Encoding Variational Bayes
vaes replace exact e-steps with learned variational inference for neural latent variables
- Neural Machine Translation by Jointly Learning to Align and Translate
attention is the neural successor to marginalizing hidden alignments in old translation models