XGBoost: A Scalable Tree Boosting System
Tianqi Chen, Carlos Guestrin2016KDD 2016
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Random Forests if you want to see why trees did not stop at bagging. The core move is to treat a new tree as a function you fit to the gradient and curvature of the loss, then make that practical under missing values, sparse features, cache limits, and distributed data. People often remember XGBoost as a Kaggle trick or a library with many knobs. That misses the lesson: the win comes from aligning the math of additive models with the mechanics of real machines. The sparsity-aware split is especially worth your time, because it turns missingness from an annoyance into a learned default direction. If you build ranking, fraud, pricing, churn, or any tabular prediction system, this paper explains why a well-engineered boosted tree can beat a neural net that looks more modern.
What to take away
- Each new tree is fit using first and second derivatives of the loss, not just residuals.
- Missing feature values get a learned default branch during split search.
- The system design matters: quantile sketches, column blocks, compression, and cache locality are part of the model's success.
Reads with
- Random Forests
bagging is the tree ensemble baseline that boosting deliberately moves beyond
- No Free Lunch Theorems for Optimization
its tabular wins are a practical reminder that inductive bias beats generic modernity