Does data drift mean the model is wrong?

No. Data drift means the input distribution changed relative to the chosen baseline. The model may still perform well, especially if the change is within patterns it learned. The next step is to check business context, label quality if available, model metrics, and whether the drift affects features the model relies on heavily.

What should be used as the baseline?

It depends on what you want to detect. Training or validation data is useful for spotting departure from what the model originally saw. A stable recent production window is better for detecting sudden operational changes. The important part is to choose deliberately, document it, and avoid refreshing it automatically in a way that absorbs real regressions.

How are PSI and KL divergence different?

PSI compares changes in bucket proportions and is popular because it is easy to put on dashboards and threshold per feature. KL divergence treats the values as probability distributions and measures how one distribution differs from another. KL can be more fragile when buckets have zero or near-zero probability, so smoothing and fixed bins are usually needed.

Are simple summaries enough for drift detection?

Often, yes. Null rate, row count, mean, median, quantiles, min and max, top categories, and category frequencies catch many production failures quickly, including logging bugs and broken joins. Distribution metrics add value when shape changes matter, but they should complement summaries rather than replace them.

Data Drift Detection

Data drift detection is the monitoring of how production feature distributions differ from a selected reference dataset. It compares current data with a baseline using statistical distances and summaries, so engineers can spot input changes that may affect models, analytics, validation rules, or downstream product behaviour before failures become obvious.

Production systems often assume that tomorrow’s data resembles the data used to build, test, or validate them. That assumption breaks when user behaviour changes, product flows are redesigned, markets shift, seasonal patterns appear, or an upstream service changes logging. Drift detection exists because these changes can leave schemas technically valid while the meaning and shape of the data have moved enough to make models or business logic unreliable.

A drift monitor takes a baseline window and a current window, then compares each feature. For numeric values it may use fixed buckets, quantiles, null rates, min and max, means, medians, spread, and tail behaviour. For categories it compares counts and frequencies. PSI looks at how bucket proportions move. KL divergence compares probability distributions, but needs care around empty or tiny buckets, often with smoothing.

The trade-off is that drift is evidence, not a verdict. A change can be harmless, expected, or even desirable, while a real problem may not cross a chosen threshold. Baselines can go stale, fresh baselines can hide regressions, and alert rules can become noisy. Bucket choices matter: if they are redefined every run, scores stop being comparable; if they are too coarse, important movement is hidden.

Engineers usually meet drift detection in scheduled monitoring jobs, feature store checks, model observability dashboards, data quality reports, and incident investigations. Practical reports show which fields moved, the direction of movement, when it began, and whether model performance, labels, conversions, or pipeline errors changed too. The best alerting is feature-specific: critical inputs deserve escalation, while minor movement in low-impact fields may only be recorded.

Go deeper

Everything DataData Drift DetectionCalculate feature drift metrics such as PSI, KL divergence, and distribution summary changes on production data.Open the lesson

Common questions

Does data drift mean the model is wrong?: No. Data drift means the input distribution changed relative to the chosen baseline. The model may still perform well, especially if the change is within patterns it learned. The next step is to check business context, label quality if available, model metrics, and whether the drift affects features the model relies on heavily.
What should be used as the baseline?: It depends on what you want to detect. Training or validation data is useful for spotting departure from what the model originally saw. A stable recent production window is better for detecting sudden operational changes. The important part is to choose deliberately, document it, and avoid refreshing it automatically in a way that absorbs real regressions.
How are PSI and KL divergence different?: PSI compares changes in bucket proportions and is popular because it is easy to put on dashboards and threshold per feature. KL divergence treats the values as probability distributions and measures how one distribution differs from another. KL can be more fragile when buckets have zero or near-zero probability, so smoothing and fixed bins are usually needed.
Are simple summaries enough for drift detection?: Often, yes. Null rate, row count, mean, median, quantiles, min and max, top categories, and category frequencies catch many production failures quickly, including logging bugs and broken joins. Distribution metrics add value when shape changes matter, but they should complement summaries rather than replace them.

Every term