Know Your Drift: Data, Concept, and Prediction Drift Explained
Model drift starts silently. Dashboards stay green, and the first signal is often a business user saying outputs feel off. By then the model has been running on stale assumptions for days. Drift detection belongs in data observability, not as a model-only add-on: inputs shift, schemas break, labels arrive late, and the output distribution wobbles before accuracy visibly drops. Model decay is common; one study found degradation in 91% of model-data pairs.
Not all drift is the same. Data drift is a change in the input distribution P(X). Concept drift is a change in the relationship between inputs and outputs, P(Y|X). Prediction drift is a shift in the output distribution. Concrete examples: a model serving an aging user base faces data drift when new users arrive with different demographics. A fraud model encounters concept drift when fraudsters switch to a new pattern. A recommendation system sees prediction drift when click-through rates change even though the input mix stays constant.
Misdiagnosing drift type wastes time. Retraining a model that actually needs a schema fix is a common failure. Each drift type calls for a different response. If a stakeholder notices drift before the platform does, the platform is monitoring the wrong thing.
Detecting Drift: Statistical Tests and Runtime Checks That Actually Work
Detect drift with statistical tests that compare distributions. The Population Stability Index (PSI) and the Kolmogorov-Smirnov (KS) test are standard tools. PSI quantifies how much a distribution has shifted; the KS test detects differences in location and shape. Run these on input features and prediction outputs.
Runtime checks on prediction logs and input schemas catch infrastructure problems. Backtest metrics on historical data spot training-serving skew and pipeline issues. Compute both production and backtest metrics as part of the monitoring pipeline.
In modern LLM and agent stacks, drift is a system-level metric. Model providers ship silent updates, retrieval indexes get re-embedded, prompts get tuned, and tool schemas evolve. A re-embedded retrieval index changes the distances between queries and documents, which shifts retrieval results even with the same model weights. Without trace-level data, you cannot separate input drift from model-quality loss.
A practical monitoring system sits alongside the prediction service. It ingests samples of input data and prediction logs, calculates metrics, and forwards them to observability platforms. Monitoring should cover model quality, drift, data quality, operational health, and fairness.
Alert Design: Setting Thresholds Without Waking Up On-Call
Alert design determines whether monitoring gets used. Drift without eval impact is a false alarm and burns on-call. Alert on the joint condition of input drift plus a measurable eval drop. That combination catches decay before it reaches decisions, customers, or downstream systems.
Set thresholds as configurable parameters, not universal values. Align them with business impact and eval metrics. For instance, you might alert when input drift crosses a threshold and the model's accuracy on a golden set falls by a fixed amount. The exact numbers depend on your domain. Alert severity should map to business impact, not to the size of the statistical shift. A small PSI shift may warrant a ticket; a large shift with a severe eval drop may page the on-call.
Instrument every production trace with embedding and eval-score logging, then compute drift metrics against a reference window. That gives the joint condition a stable basis.
Define an escalation path to human review when the joint condition fires. Trigger an investigation playbook: check the input schema, review recent data pipeline runs, compare prediction distributions, and inspect eval logs. If the evidence points to concept drift, escalate to a model retraining decision. If it points to a schema change, fix the pipeline first.
From Drift to Action: Integration, Retraining, and Human Oversight
For Prometheus and Grafana, expose drift metrics through a sidecar that samples raw inputs and predictions. Raw samples matter: comparing distributions requires the original feature values, not just aggregated logs. The sidecar keeps drift detection separate from the serving path, so monitoring never adds latency to predictions.
When the joint condition fires, the escalation path lands on a human. Choose between retraining and rolling back to a previous model version. Drift severity and eval performance guide the choice, but a person makes the call. Rolling back is often faster and safer than retraining when the evidence points to a deployment issue.
Set retraining policies based on drift severity and eval results. There is no universal schedule. Some models need weekly updates; others run for months. The monitoring data tells you when to act.
