Model evaluation and metrics (advanced) quiz

Question 1

Why can a model with high AUC-ROC still perform poorly in a highly imbalanced fraud-detection setting, and what metric better reflects ranking quality there?

Accepted Answer

ROC's FPR uses the large negative count as denominator and is insensitive to many false positives; AUC-PR is more informative

Answer

AUC-ROC ignores true positives, so it should be replaced by accuracy

Answer

AUC-ROC always equals AUC-PR under imbalance

Answer

Imbalance has no effect on AUC-ROC interpretation

Question 2

What is the key difference between calibration and discrimination, and can a model be perfectly calibrated yet useless?

Accepted Answer

A model outputting the base rate is perfectly calibrated but has AUC=0.5, hence useless for ranking

Answer

A perfectly calibrated model must have AUC=1

Answer

Calibration and discrimination are the same property

Answer

Discrimination measures probability accuracy and calibration measures ranking

Question 3

How does the Expected Calibration Error (ECE) work, and what is a known weakness of it?

Accepted Answer

ECE averages |confidence - accuracy| over bins but is sensitive to binning and can hide intra-bin compensation

Answer

ECE is bin-independent and always unbiased

Answer

ECE measures only ranking ability

Answer

ECE is a proper scoring rule

Question 4

Why is log loss (cross-entropy) called a 'proper scoring rule' and what behavior does that property guarantee?

Accepted Answer

It is strictly proper: expected loss is uniquely minimized by reporting true probabilities

Answer

It is proper because it only depends on the predicted class

Answer

Proper means it is bounded between 0 and 1

Answer

Proper means it ignores calibration

Question 5

In multi-class classification, what is the difference between macro, micro, and weighted F1, and when would macro and micro diverge most?

Accepted Answer

Micro pools counts (favors frequent classes), macro weights classes equally; they diverge under imbalance with weak minority classes

Answer

Macro and micro F1 are always equal

Answer

Weighted F1 ignores class support

Answer

Macro F1 equals accuracy in all cases

Question 6

What does the Brier score decompose into via the Murphy decomposition, and what does each component mean?

Accepted Answer

Brier = Reliability - Resolution + Uncertainty

Answer

Brier = bias + variance + noise

Answer

Brier = precision - recall + base rate

Answer

Brier = Resolution - Reliability + Uncertainty

Question 7

When evaluating a ranking/recommendation system, what advantage does NDCG have over MAP?

Accepted Answer

NDCG supports graded relevance with positional discounting and ideal-DCG normalization; MAP assumes binary relevance

Answer

MAP supports graded relevance while NDCG does not

Answer

They are mathematically identical

Answer

NDCG ignores rank position

Question 8

Why can optimizing for a single threshold-based metric like accuracy be misleading, and what does the Matthews Correlation Coefficient (MCC) add?

Accepted Answer

MCC uses all four confusion-matrix cells and is robust to imbalance, ranging -1 to +1

Answer

MCC ignores true negatives

Answer

MCC equals accuracy under all distributions

Answer

MCC only works for regression

Question 9

What is the difference between BLEU and chrF for evaluating generated text, and why is chrF often preferred for morphologically rich languages?

Accepted Answer

BLEU is word n-gram precision; chrF uses character n-gram F-score, better capturing morphology

Answer

chrF is word-level and BLEU is character-level

Answer

Both ignore n-grams entirely

Answer

chrF requires reference-free evaluation

Question 10

Why are embedding-based metrics like BERTScore considered improvements over BLEU/ROUGE, and what is a major limitation?

Accepted Answer

BERTScore uses contextual-embedding similarity to capture semantics but depends on the encoder and still needs references

Answer

BERTScore uses exact n-gram overlap like BLEU

Answer

BERTScore is reference-free and encoder-agnostic

Answer

BERTScore measures only fluency, not meaning

Question 11

In LLM-as-a-judge evaluation, what biases must be controlled, and name two mitigation techniques.

Accepted Answer

Position and verbosity bias exist; mitigate via order-swapping and rubric/reference-based scoring

Answer

LLM judges have no measurable biases

Answer

The only bias is tokenization, fixed by lowercasing

Answer

Bias is removed simply by raising temperature

Question 12

What is the purpose of a bootstrap confidence interval for a metric, and how does the paired bootstrap improve model comparison?

Accepted Answer

Bootstrap resamples to estimate a metric's sampling distribution; paired bootstrap reuses the same instances for both models to reduce variance

Answer

Bootstrap requires the data to be Gaussian

Answer

Paired bootstrap uses different test sets per model

Answer

Bootstrap can only estimate the mean, never CIs

Question 13

How does threshold-moving differ from cost-sensitive learning, and how do you pick an optimal threshold for an asymmetric cost matrix?

Accepted Answer

Threshold-moving adjusts the cutoff post hoc; optimal threshold is C_FP/(C_FP+C_FN)

Answer

Optimal threshold is always 0.5 regardless of costs

Answer

Cost-sensitive learning only changes the threshold, not training

Answer

Threshold should equal the positive base rate

Question 14

What is the difference between Platt scaling and isotonic regression for calibration, and when is each appropriate?

Accepted Answer

Platt fits a sigmoid (good for little data); isotonic fits a monotonic step function (flexible but data-hungry)

Answer

Platt is non-parametric and isotonic is parametric

Answer

Both assume a Gaussian score distribution

Answer

Isotonic can produce non-monotonic mappings

Question 15

Why is temperature scaling popular for calibrating deep neural networks, and what does it leave unchanged?

Accepted Answer

It scales logits by one scalar T, preserving argmax/accuracy while improving calibration

Answer

It changes the predicted class to improve calibration

Answer

It retrains all network weights

Answer

It only works for regression outputs

Question 16

What is the multiple-comparisons problem in model evaluation, and how does it inflate false discoveries when picking the 'best' of many models?

Accepted Answer

The max score over many models is optimistically biased; use a one-time test set, nested CV, or multiple-comparison correction

Answer

Evaluating many models has no statistical cost

Answer

Bonferroni increases false positives

Answer

Validation overfitting only affects training metrics

Question 17

Why should you use stratified or grouped/time-series cross-validation instead of plain k-fold in certain scenarios?

Accepted Answer

Stratified preserves class balance, grouped prevents group leakage, time-series respects temporal order to avoid look-ahead leakage

Answer

Plain k-fold is always optimal

Answer

Time-series CV should shuffle data randomly

Answer

Grouped CV splits a group across folds for robustness

Question 18

What does it mean for a regression metric to be scale-dependent versus scale-independent, and give an example of each plus a pitfall of MAPE.

Accepted Answer

RMSE/MAE are scale-dependent; MAPE is scale-independent but blows up near zero and is asymmetric

Answer

MAPE is well-defined even when actuals are zero

Answer

R^2 is scale-dependent

Answer

MAPE symmetrically penalizes over- and under-prediction

Question 19

Why does RMSE penalize large errors more than MAE, and what does the gap between them tell you about the error distribution?

Accepted Answer

RMSE squares errors so outliers dominate; a large RMSE/MAE gap signals high error-magnitude variance

Answer

MAE penalizes large errors more than RMSE

Answer

RMSE is always less than MAE

Answer

The RMSE/MAE ratio reveals nothing about error spread

Question 20

What is a reliability diagram, and how do you read overconfidence from it?

Accepted Answer

Points below the diagonal (accuracy < confidence) indicate overconfidence

Answer

Points above the diagonal indicate overconfidence

Answer

The diagonal represents random guessing

Answer

Reliability diagrams plot precision vs recall

Question 21

When comparing two classifiers on the same test set, why might McNemar's test be preferred over comparing accuracies directly?

Accepted Answer

McNemar tests symmetry of the discordant (disagreement) pairs, giving a valid paired significance test

Answer

McNemar uses the full confusion matrix of both models equally

Answer

McNemar requires independent test sets

Answer

It compares raw accuracies with a t-test

Question 22

In retrieval-augmented generation (RAG) evaluation, what do 'faithfulness' and 'answer relevance' measure, and why are both needed?

Accepted Answer

Faithfulness = answer grounded in context; answer relevance = answer addresses the query; they are orthogonal

Answer

They both measure retrieval latency

Answer

Faithfulness measures fluency and relevance measures grammar

Answer

Only one of them is needed since they are equivalent

Question 23

What is the difference between context precision and context recall in RAG retrieval evaluation?

Accepted Answer

Context recall = needed facts retrieved; context precision = retrieved chunks are relevant/well-ranked

Answer

Context precision measures whether all needed facts were retrieved

Answer

Both measure generation fluency

Answer

They are identical to BLEU and ROUGE

Question 24

How does Cohen's kappa adjust raw agreement, and why can it be paradoxically low even with high observed agreement?

Accepted Answer

Kappa = (p_o - p_e)/(1 - p_e); under skewed marginals high agreement can still give low kappa

Answer

Kappa ignores chance agreement entirely

Answer

Kappa is always higher than raw agreement

Answer

Kappa can never be negative or below zero

Question 25

Why is R^2 alone insufficient for evaluating regression models, and what does adjusted R^2 fix?

Accepted Answer

In-sample R^2 never decreases with more predictors; adjusted R^2 penalizes added parameters

Answer

R^2 decreases when irrelevant features are added

Answer

Adjusted R^2 always equals R^2

Answer

R^2 cannot be negative on any data

Question 26

What is the difference between AIC and BIC for model selection, and how do their penalties differ asymptotically?

Accepted Answer

AIC penalty is 2k, BIC is k*ln(n); BIC penalizes complexity more and is consistent

Answer

AIC penalty is k*ln(n), BIC penalty is 2k

Answer

Both use identical penalties

Answer

BIC always selects larger models than AIC

Question 27

In a precision-recall tradeoff, what is the F-beta score and how does beta change the emphasis?

Accepted Answer

F_beta weights recall more as beta grows (>1) and precision more as beta shrinks (<1)

Answer

beta < 1 emphasizes recall

Answer

beta has no effect on the weighting

Answer

F-beta is the arithmetic mean of P and R

Question 28

What is prediction/data drift versus concept drift, and which evaluation strategy detects concept drift?

Accepted Answer

Concept drift changes P(y|X) and needs labeled feedback (e.g., rolling metric tests) to detect

Answer

Concept drift is a change in P(X) only and needs no labels

Answer

Prediction drift requires labels but concept drift does not

Answer

Both are detectable from inputs alone

Question 29

Why is PSI (Population Stability Index) used in production, and what thresholds typically signal concern?

Accepted Answer

PSI sums (a%-e%)*ln(a%/e%); <0.1 stable, 0.1-0.25 moderate, >0.25 major shift

Answer

PSI > 0.25 means no shift

Answer

PSI is a calibration metric for probabilities

Answer

PSI requires labels to compute

Question 30

What are group fairness metrics like demographic parity and equalized odds, and why can they be mutually unsatisfiable?

Accepted Answer

When base rates differ, calibration, equalized odds, and demographic parity generally cannot all hold simultaneously

Answer

All fairness metrics can always be satisfied at once

Answer

Demographic parity requires equal TPR and FPR

Answer

Equalized odds means equal positive prediction rates

Test yourself