Cheat sheet · Data & MLOps

Model Monitoring & Drift cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

Model monitoring is the practice of continuously tracking a deployed model's data, predictions, and business outcomes to detect degradation early. Drift (covariate, concept, label/prior, and prediction drift) is the primary cause of silent decay, and a mature monitoring stack combines statistical detection, ground-truth-aware performance metrics, label-free performance estimation, and automated alerting/retraining with validation gates.

Taxonomy of Drift

Covariate/data drift: input distribution P(X) changes while P(Y|X) stays fixed (e.g., new user demographics). The learned mapping can still be valid, but inputs fall outside the training support, so extrapolation risk rises.
Concept drift: the relationship P(Y|X) changes - the mapping the model learned is now wrong (e.g., fraud tactics evolve, post-COVID demand). This is what directly hurts accuracy and usually requires retraining.
Label/prior drift: P(Y) shifts (target class balance changes), often correctable via prior/prevalence adjustment (recalibrating predicted probabilities) without retraining features.
Prediction drift: distribution of model outputs P(Y-hat) shifts - a useful proxy you can monitor in real time WITHOUT ground truth, but it cannot by itself distinguish harmless covariate drift from harmful concept drift.
Drift dynamics: sudden (regime break), gradual, incremental, and recurring/seasonal - the detection window and reference strategy differ for each.
Upstream data-quality issues (schema changes, null spikes, unit/encoding changes, broken joins, stale features) masquerade as drift and are far more common in practice - triage these first.

Detecting Data Drift (no labels needed)

Numeric features: Population Stability Index (PSI), Kolmogorov-Smirnov (KS) two-sample test, Wasserstein/Earth-Mover distance, Jensen-Shannon (JS) divergence.
Categorical features: PSI, Chi-square test, JS/KL divergence over category proportions; also watch for unseen categories.
PSI rule of thumb: <0.1 stable, 0.1-0.25 moderate shift, >0.25 significant drift. Popular because it is a single interpretable effect-size number with stable thresholds, unlike p-values that explode with sample size.
Multivariate drift: a domain classifier (train a model to distinguish reference vs current; ROC-AUC ~ 0.5 means indistinguishable/no drift) or PCA-reconstruction error captures correlation shifts that per-feature tests miss.
Embeddings/unstructured data: monitor distances in embedding space (Maximum Mean Discrepancy, cosine distance to a reference centroid) or use density/out-of-distribution detectors.
Compare against a fixed, representative reference window (training set or a validated production baseline). A rolling reference that moves with the data will mask gradual drift entirely.

Detecting Concept Drift & Performance Decay

Direct performance tracking (AUC, F1, log-loss, RMSE, calibration) is the gold standard but needs ground-truth labels, which arrive with delay or never.
Label-delay strategies: monitor leading proxies (prediction drift, confidence/entropy distribution) now, then backfill true metrics once delayed labels land.
Streaming detectors on the online error stream: DDM/EDDM (error-rate based), ADWIN (adaptive windowing), and the Page-Hinkley test flag change points.
Label-free performance estimation: CBPE (Confidence-Based Performance Estimation) for classification and DLE (Direct Loss Estimation) for regression infer metrics from model outputs/confidence; both require a well-calibrated reference and assume no concept drift, so pair them with drift checks.
Track calibration drift separately: a model can preserve ranking (stable AUC) while probabilities become miscalibrated - monitor ECE and reliability curves.
Segment/slice monitoring: aggregate metrics can look fine while a key cohort (region, device, new feature value, new customer segment) silently collapses - slice by business-relevant dimensions.

Monitoring Stack & Metric Layers

Four layers: (1) service/operational health (latency, throughput, error rate, resource use), (2) data quality (schema, nulls, ranges, freshness), (3) data/prediction drift, (4) model performance & business KPIs.
Business/outcome metrics close the loop - a statistically 'fine' model that stops driving conversions/revenue is still failing.
Log inputs, outputs, predicted probabilities, model version, feature values, and request id for every prediction to enable replay, root-cause analysis, and a later label join.
Tooling: Evidently, NannyML, WhyLabs/whylogs, Arize, Fiddler, Aporia for ML-specific monitoring; Prometheus/Grafana + OpenTelemetry for ops; Great Expectations / Soda / dbt tests for data quality.
Alert on sustained, aggregated signals (e.g., drift persisting over N windows) with hysteresis - not single noisy points - to control false alarms.
Serve features from the same definitions used in training (a feature store / shared transforms) to prevent training-serving skew, a leading cause of 'drift' that is actually a pipeline bug.

LLM / GenAI Monitoring (2026)

No single accuracy metric: monitor task-specific quality via LLM-as-judge (with bias/position controls), reference-based scores where you have golden answers, and sampled human/thumbs feedback.
Track hallucination/faithfulness (grounding to retrieved context), toxicity/safety, PII leakage, refusal rate, and prompt-injection/jailbreak attempts.
Embedding drift on inputs detects shifting user intent; output drift (length, format-validity, JSON/schema validity, tool-call success rate) catches silent regressions after a prompt or model swap.
RAG-specific: monitor retrieval quality (context relevance, recall, chunk hit rate) separately from generation - degradation is often in retrieval, not the LLM.
Cost & latency are first-class: tokens/request, cost/request, cache-hit rate, and p95/p99 latency directly affect viability.
Guardrails/online evals run inline (block or flag bad outputs) while offline eval suites run on sampled/traced traffic for trend tracking and regression gating in CI.

Response: Alerting, Retraining & Governance

Define a runbook per alert: triage data quality first, then drift, then concept change - avoid retraining on what is actually a broken pipeline.
Retraining triggers: scheduled (cadence), performance-threshold, or drift-threshold based; threshold-triggered is efficient but needs cooldowns/guards against thrashing.
Always validate a retrained model on a holdout AND via shadow/canary before full rollout; never auto-promote on a drift alert alone.
Shadow mode (run the new model alongside the old with no live impact) and A/B or canary rollout limit blast radius and enable fast rollback.
Maintain a feedback/label pipeline: capture ground truth, delayed outcomes, and human corrections to feed both monitoring and retraining datasets.
Governance: version data, features, model, and config; record lineage, evals, and approvals for auditability and reproducible rollback.

Common pitfalls

Equating any data drift with model failure - covariate drift can be harmless if P(Y|X) is unchanged; act when performance, calibration, or a meaningful slice degrades.
Monitoring only aggregate accuracy and missing cohort/segment collapse hidden by averages.
Using a rolling reference window so the baseline drifts with the data, masking gradual drift.
Running per-feature univariate tests only and missing multivariate/correlation drift.
Ignoring label delay - assuming live accuracy is computable when labels arrive days/weeks later; build proxy and label-free estimated metrics.
Treating training-serving skew or schema/unit changes as 'concept drift' and retraining instead of fixing the pipeline.
Relying on p-value tests (KS/Chi-square) on huge samples - they flag trivial, practically irrelevant differences; pair with effect-size thresholds (PSI/Wasserstein).
Alert fatigue from single-point thresholds with no hysteresis or persistence requirement.
Auto-retraining/auto-promoting on a drift signal without validation, shadow testing, or a rollback path.
Forgetting calibration and business KPIs - a well-ranked but miscalibrated or non-revenue-driving model still fails.
Trusting label-free estimators (CBPE/DLE) blindly - they assume no concept drift and a calibrated reference, so they degrade exactly when concept drift hits.

Interview tips

Crisply distinguish covariate vs concept vs label vs prediction drift, with a concrete example and the correct response for each.
Expect 'How do you monitor a model when labels are delayed or unavailable?' - answer with prediction drift, proxy metrics, and label-free performance estimation (CBPE for classification, DLE for regression; NannyML).
Know specific metrics and thresholds (PSI bands, KS, Wasserstein, JS divergence) and when each fits; explain why PSI's effect-size framing beats raw p-values.
Explain the multivariate drift trick (domain classifier, AUC ~ 0.5 = no drift) to show depth beyond univariate tests.
Walk the full monitoring stack: ops, data quality, drift, performance, business - and state your triage order when an alert fires (data quality first).
Describe a retraining/response strategy: trigger types, shadow/canary deployment, validation gates, cooldowns to avoid thrashing.
For senior/GenAI roles, cover LLM monitoring: LLM-as-judge, hallucination/faithfulness, RAG retrieval quality, embedding/output drift, cost & latency, online guardrails vs offline evals.
Name concrete tooling (Evidently, NannyML, WhyLabs, Arize, Fiddler + Prometheus/Grafana) to signal current, hands-on experience.
Tie it back to training-serving consistency and feature stores - interviewers probe whether you know most 'drift' is really a pipeline bug.

Read the full lesson