Random Forests
Leo Breiman2001Machine Learning 2001
Read it on doi.org(opens in a new tab)Why this one
Read this before XGBoost. Breiman gives you the clean version of an idea that still wins boring production fights: take unstable trees, decorrelate them with random feature choices and bootstrap samples, then average away their noise. What people get wrong is treating random forests as an old bag of heuristics. The paper is really about strength and correlation: each tree should be better than chance, but not too much like its neighbors. That is a useful mental model far beyond trees, from model ensembles to A/B metric smoothing. If you work on tabular data, this is the baseline that keeps neural networks honest. It handles messy features, nonlinear interactions, and little preprocessing, while giving you out of bag error as a built in sanity check. Worth your evening because it teaches a kind of practical statistics you can ship.
What to take away
- Forest quality depends on strong individual trees with low correlation between their errors.
- Out of bag samples give a built in validation estimate without setting aside a separate test split.
- Random feature selection at each split is the key move that decorrelates trees without making them weak.
Reads with
- XGBoost: A Scalable Tree Boosting System
boosting is the next tabular workhorse, trading forest averaging for sequential error correction
- Dropout: A Simple Way to Prevent Neural Networks from Overfitting
the same ensemble instinct reappears as random subnetworks averaged inside one neural net