What does data drift mean in a production ML system?

Data drift means the statistical shape of production inputs has moved away from a trusted reference set, such as training data or a stable production window. It is not proof that the model is wrong, but it is a signal that the process creating the data may have changed and should be checked.

When should I use PSI instead of KL divergence?

PSI is often better for operational dashboards and alerting because it compares bucket proportions in a way that is relatively easy to explain. KL divergence is useful when you want a distribution distance measure, but it is more sensitive to empty or tiny buckets and usually needs smoothing to avoid misleading results.

What should a useful drift alert include?

A useful drift alert should name the affected feature, show the direction and size of the change, identify when it began, and relate it to downstream quality or model behaviour where possible. Severity should depend on feature importance and impact, not just whether a metric crossed a generic threshold.

Data Quality & Monitoring

08.04 · Walkthrough

Data Drift Detection

Calculate feature drift metrics such as PSI, KL divergence, and distribution summary changes on production data.

Data drift detection compares production feature distributions with a trusted baseline so changes in inputs are visible before they damage downstream behaviour. Practical monitoring combines PSI, KL divergence, and simple summaries such as null rates, quantiles, and category frequencies, with fixed buckets, deliberate baselines, and feature-specific alert severity.

What this lesson answers

how to calculate PSI for feature drift
KL divergence for production data drift monitoring
which summaries detect distribution changes in production

Notes

Data drift detection is the practice of comparing the statistical shape of production data against a trusted baseline, such as training data, validation data, or a recent “known good” production window. The goal is to catch cases where the input features your pipelines and models depend on have changed enough to affect downstream behavior. Drift does not automatically mean something is broken; it means the data generating process may have shifted.

Common questions

What does data drift mean in a production ML system?: Data drift means the statistical shape of production inputs has moved away from a trusted reference set, such as training data or a stable production window. It is not proof that the model is wrong, but it is a signal that the process creating the data may have changed and should be checked.
When should I use PSI instead of KL divergence?: PSI is often better for operational dashboards and alerting because it compares bucket proportions in a way that is relatively easy to explain. KL divergence is useful when you want a distribution distance measure, but it is more sensitive to empty or tiny buckets and usually needs smoothing to avoid misleading results.
What should a useful drift alert include?: A useful drift alert should name the affected feature, show the direction and size of the change, identify when it began, and relate it to downstream quality or model behaviour where possible. Severity should depend on feature importance and impact, not just whether a metric crossed a generic threshold.

Short definition: what is Data Drift Detection?

08.03 · Data Quality & MonitoringContract Tests for Pipelines 08.05 · Data Quality & MonitoringLabel and Concept Drift