Concept drift versus data drift: choose the right response

Data drift changes the distribution of inputs. Concept drift changes the relationship between inputs and the target. They can occur together, but they call for different evidence. Input-only monitoring can detect neither the true relationship nor model accuracy without labels.

Two contrived scenarios

The local deployment lab begins with test accuracy 72% on its synthetic risk task.

from deployment_cases import concept_drift_case

result = concept_drift_case()
assert result["concept_drift_accuracy_after_label_rule_flip"] < result["base_accuracy"]
print(result)

In the data-drift scenario, monthly spend shifts but labels continue to follow the authored relationship; accuracy against its deterministic rule is 95.3%. In the concept scenario, the label rule is deliberately flipped while inputs stay the same; accuracy falls to 28%.

These mechanisms are known only because the fixture constructs them. In production, a performance drop does not reveal its cause automatically. Investigate policy, label definition, population, data pipeline and operational behavior.

Choose evidence by question

Input histograms, missingness and category shares support data-drift investigation. Mature labels, calibration and slice metrics support concept/performance monitoring. A target-definition change may look like concept drift but is actually a contract change requiring relabeling and re-evaluation.

If input drift occurs without performance loss, continued monitoring may be enough. If performance falls without obvious input drift, check hidden or unobserved variables, process changes and label quality. Do not retrain simply because one drift statistic crosses a generic threshold.

Respond through a candidate gate

Collect fresh representative labels, retrain as a candidate, compare against baseline and test critical slices. Release through normal schema, artifact, canary and rollback gates. Preserve the failing model’s evidence for diagnosis.

The Data Science course connects drift diagnosis to a measured response rather than an automatic pipeline reaction.

Exercise

Simulate covariate shift, label prevalence shift and conditional label-rule shift. For each, list observable monitors, missing evidence and the response you would authorize.

Continue learning

This article is part of the Model deployment and MLOps sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Monitor model input drift without treating every alert as failure.
• Continue with Design delayed-label monitoring for a deployed model.

Reference: scikit-learn probability calibration guide.