ML Observability quiz

Question 1

What is ML observability and how does it differ from traditional software monitoring?

Accepted Answer

ML observability monitors model degradation and data drift that cause silent failures not caught by traditional uptime/error monitoring

Answer

ML observability is identical to APM for Python services

Answer

ML observability only applies to computer vision models

Answer

ML monitoring is only needed when model accuracy drops below 50%

Question 2

What is the Population Stability Index (PSI) and how do you interpret its values?

Accepted Answer

PSI < 0.10 no drift; 0.10-0.20 minor; > 0.20 significant drift requiring investigation or retraining

Answer

PSI < 0.50 is acceptable; > 0.50 indicates system failure

Answer

PSI measures model accuracy not distribution shift

Answer

PSI applies only to categorical features

Question 3

What is concept drift and how does it differ from data drift?

Accepted Answer

Concept drift changes P(Y|X) — the input-output relationship; data drift changes P(X) — the input distribution

Answer

They are identical terms for the same phenomenon

Answer

Data drift affects only deep learning models; concept drift affects classical ML

Answer

Concept drift is easier to detect because it shows in accuracy metrics immediately

Question 4

What is the delayed labels problem in ML monitoring and how do you address it?

Accepted Answer

Use proxy metrics, prediction distribution monitoring, and retrospective label joining when labels arrive with delay

Answer

Ignore labels and only monitor prediction distributions for all use cases

Answer

Delayed labels only affect classification models, not regression

Answer

Retrain models immediately when labels are delayed by more than 24 hours

Question 5

What are fairness metrics in ML and how do you monitor them in production?

Accepted Answer

Demographic Parity (equal positive rates), Equal Opportunity (equal TPR), Equalized Odds (equal TPR+FPR) across protected groups

Answer

Fairness is measured solely by the model's overall accuracy across all groups

Answer

Fairness metrics only apply to criminal justice applications

Answer

Equal precision across groups always satisfies all fairness criteria

Question 6

What is NannyML and what problem does it solve without requiring labels?

Accepted Answer

Estimates model performance without labels using CBPE — modeling confidence-accuracy relationships from validation data

Answer

A monitoring tool that requires 100% labeled production data to function

Answer

A data quality tool for detecting schema changes in feature stores

Answer

An alternative to SHAP for post-hoc model explanation

Question 7

What is the champion-challenger pattern in ML model management?

Accepted Answer

Champion serves most traffic; challenger serves a small slice for online comparison before promotion

Answer

Champion and challenger run in shadow mode without serving real users

Answer

Only one model (champion) runs at a time; challenger is offline

Answer

The challenger always replaces the champion after one week

Question 8

What is model card documentation and why is it considered essential for ML governance?

Accepted Answer

Structured documentation covering model purpose, performance across groups, limitations, and ethical considerations for transparency

Answer

A card game training dataset for ML education

Answer

A Kubernetes ConfigMap storing model hyperparameters

Answer

A credit card analogy for ML model billing

Question 9

What is permutation feature importance and how does it differ from SHAP for monitoring model behavior?

Accepted Answer

Measures global performance degradation from shuffling each feature; differs from SHAP which explains individual predictions locally

Answer

Permutation importance and SHAP produce identical results by mathematical equivalence

Answer

Permutation importance only works for decision tree models

Answer

Shuffling features randomly improves model robustness rather than measuring importance

Question 10

What is calibration in ML models and how do you monitor calibration drift in production?

Accepted Answer

A calibrated model's confidence scores match true positive rates; monitor ECE on labeled samples and recalibrate when drift detected

Answer

Calibration means the model achieves > 90% accuracy on the test set

Answer

Calibration only matters for ranking models, not classification

Answer

A well-calibrated model always predicts exactly 50% for all inputs

Test yourself