Model monitoring and drift quiz

Question 1

What is the difference between data drift and concept drift?

Accepted Answer

Data drift changes P(X); concept drift changes P(Y|X)

Answer

Data drift changes P(Y|X); concept drift changes P(X)

Answer

They are identical terms for label noise

Answer

Both refer only to changes in the target distribution

Question 2

Why is the Population Stability Index (PSI) widely used for monitoring feature and score drift?

Accepted Answer

PSI > 0.25 typically signals significant population shift

Answer

PSI > 0.25 means the distributions are identical

Answer

PSI is only defined for continuous targets

Answer

PSI requires labeled outcomes to compute

Question 3

What is training-serving skew and what are its common causes?

Accepted Answer

Mismatch between feature computation at training vs inference time

Answer

A change in label distribution over time

Answer

The model overfitting the training set

Answer

Latency between request and prediction

Question 4

How does the Kolmogorov-Smirnov (KS) test detect drift in a continuous feature?

Accepted Answer

It compares empirical CDFs via the maximum vertical distance

Answer

It compares means using a t-test

Answer

It counts category frequency mismatches

Answer

It measures correlation between features

Question 5

With very large monitoring windows, why can statistical drift tests like KS or chi-square become misleading?

Accepted Answer

Large samples make tests over-sensitive, flagging trivial differences

Answer

Large samples reduce power and miss real drift

Answer

The tests stop working above 10,000 rows

Answer

p-values become unreliable only with categorical data

Question 6

What is the ground-truth lag (label delay) problem in model monitoring, and how do teams cope?

Accepted Answer

Use input/prediction drift as immediate proxies until labels arrive

Answer

Stop monitoring until labels are available

Answer

Assume accuracy is constant between label arrivals

Answer

Generate synthetic labels with the model itself

Question 7

What is prediction drift and why monitor it separately from input drift?

Accepted Answer

A change in the distribution of model outputs over time

Answer

A change only in the input feature means

Answer

The model predicting slower than its SLA

Answer

Drift in the training labels

Question 8

Explain the difference between covariate shift, label shift (prior probability shift), and concept shift.

Accepted Answer

Covariate: P(X) shifts; label: P(Y) shifts; concept: P(Y|X) shifts

Answer

All three are synonyms for accuracy decay

Answer

Covariate shifts P(Y|X); concept shifts P(X)

Answer

Label shift means feature drift only

Question 9

How does the ADWIN (ADaptive WINdowing) algorithm detect drift in streaming data?

Accepted Answer

Splits an adaptive window and tests sub-window means via a Hoeffding bound

Answer

Uses a fixed window and a t-test

Answer

Requires labels and recomputes accuracy each step

Answer

Trains a new model on every incoming sample

Question 10

What is the Drift Detection Method (DDM) and what signal does it monitor?

Accepted Answer

It tracks the online error rate and flags drift when it exceeds p_min+3s_min

Answer

It tracks input feature variance only

Answer

It is an unsupervised embedding monitor

Answer

It measures latency degradation

Question 11

Why are KL divergence and Jensen-Shannon divergence treated differently when measuring distribution drift?

Accepted Answer

JSD is symmetric and bounded; KL is asymmetric and can be infinite

Answer

KL is symmetric and bounded; JSD is unbounded

Answer

They are mathematically identical

Answer

Both are undefined for discrete distributions

Question 12

For monitoring drift in high-dimensional embeddings (e.g., from an LLM or image model), why is per-feature univariate testing insufficient?

Accepted Answer

It misses joint/correlation changes and inflates multiple-comparison errors

Answer

It is computationally impossible

Answer

It only works for images, not text

Answer

Univariate tests detect all multivariate drift perfectly

Question 13

What is a classifier two-sample test (domain classifier) for drift detection?

Accepted Answer

AUC near 0.5 means no drift; significantly higher AUC means drift

Answer

AUC near 1.0 means no drift

Answer

It requires the production model's labels

Answer

It only works on a single feature at a time

Question 14

Why might a model's accuracy stay flat while the business KPI it supports degrades, and what does this imply for monitoring?

Accepted Answer

Global accuracy can mask segment, calibration, or tail degradation affecting KPIs

Answer

Accuracy and KPI are mathematically equivalent

Answer

Business KPIs are irrelevant to ML monitoring

Answer

KPI changes always show up first in accuracy

Question 15

What is calibration drift and how can you monitor it without immediate labels?

Accepted Answer

Track predicted-probability distribution and mean score vs expected base rate

Answer

Calibration cannot be monitored at all without labels

Answer

Use accuracy as a perfect calibration proxy

Answer

Calibration only matters at training time

Question 16

How does the Page-Hinkley test detect drift, and where is it typically applied?

Accepted Answer

It is a sequential test on cumulative deviation from a running mean

Answer

It compares two static histograms

Answer

It requires retraining at each step

Answer

It only detects variance changes, not mean shifts

Question 17

What is the Wasserstein (earth mover's) distance and what advantage does it offer over bin-based divergences for drift?

Accepted Answer

It measures the cost of moving mass and stays meaningful with little overlap

Answer

It requires identical support and explodes otherwise

Answer

It only works on categorical features

Answer

It is identical to KL divergence

Question 18

In monitoring, what is the distinction between data-quality issues and genuine data drift?

Accepted Answer

Quality issues are pipeline defects; drift is a real population change

Answer

They are the same and handled identically

Answer

Drift is always fixed by data engineering

Answer

Quality issues always require retraining the model

Question 19

What is a feedback loop in a deployed model, and how can it corrupt monitoring signals?

Accepted Answer

Predictions influence future data/labels, biasing observed metrics

Answer

It always improves monitoring accuracy

Answer

It only affects training, never serving

Answer

It is the same as concept drift

Question 20

When should retraining be triggered by drift detection versus on a fixed schedule?

Accepted Answer

Trigger-based for sporadic shifts; scheduled for steady evolution; often both

Answer

Always retrain on every new sample

Answer

Never retrain once deployed

Answer

Only schedule-based retraining is ever valid

Question 21

Why can blindly retraining on the most recent data in response to drift make a model worse?

Accepted Answer

Recent data may be buggy, biased, or transient, degrading the new model

Answer

Recent data is always cleaner than old data

Answer

Retraining can never reduce performance

Answer

More data always guarantees improvement

Question 22

What is a champion-challenger (shadow) deployment and how does it support safe model updates?

Accepted Answer

Challengers score live traffic in shadow for comparison without affecting users

Answer

The challenger immediately replaces the champion

Answer

Both serve 50% of users by default

Answer

It is a technique only for A/B label collection

Question 23

How do you monitor a model in production when ground-truth labels are essentially never available (e.g., unsupervised anomaly detection)?

Accepted Answer

Use drift proxies, alert-rate stability, and periodic human audits

Answer

Stop monitoring since labels are absent

Answer

Assume the model is correct indefinitely

Answer

Only latency monitoring is possible

Question 24

What is the role of a reference (baseline) dataset window choice in drift detection, and what are the trade-offs?

Accepted Answer

Fixed baseline catches training deviation; sliding adapts but hides slow drift

Answer

Sliding baselines never miss any drift

Answer

The baseline choice has no effect on alerts

Answer

A baseline is unnecessary for drift detection

Question 25

Why is monitoring drift on the most important features (by model importance) often prioritized over monitoring all features equally?

Accepted Answer

Drift impact scales with feature importance, so prioritize key drivers

Answer

All features always have equal impact on predictions

Answer

Low-importance features cause the most failures

Answer

Feature importance is irrelevant to drift impact

Question 26

What specific monitoring concerns arise for LLM / GenAI applications that do not apply to classic tabular models?

Accepted Answer

Hallucination, toxicity, prompt-injection, relevance, and cost via judges/guardrails

Answer

Only token throughput needs monitoring

Answer

Standard accuracy fully covers LLM monitoring

Answer

LLMs require no monitoring once fine-tuned

Question 27

For a RAG system, what does retrieval drift mean and why monitor it?

Accepted Answer

Decline in relevance of retrieved context that caps generation quality

Answer

A change in the LLM weights only

Answer

Increased token cost with no quality impact

Answer

Drift in the user interface layout

Question 28

What is the difference between sudden, gradual, incremental, and recurring concept drift?

Accepted Answer

They describe how fast/in what pattern P(Y|X) changes (abrupt, slow, continuous, cyclic)

Answer

They are four names for data-quality bugs

Answer

They differ only in feature count

Answer

They all require identical handling

Question 29

How can you distinguish seasonal patterns from genuine drift to avoid false retraining triggers?

Accepted Answer

Compare against a same-season baseline or monitor deseasonalized residuals

Answer

Always retrain whenever any deviation appears

Answer

Ignore seasonality since it is the same as drift

Answer

Use only a 1-day reference window

Question 30

What is importance weighting (e.g., via density-ratio estimation) used for under covariate shift?

Accepted Answer

Reweight training samples by p_test(x)/p_train(x) to correct covariate shift

Answer

Relabel the test set automatically

Answer

Correct concept drift in P(Y|X) directly

Answer

Increase the learning rate during drift

Test yourself