Cheat sheet · Machine Learning

Model evaluation metrics cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

A dense reference to choosing, computing, and reasoning about evaluation metrics across classification, regression, ranking/probability, and generative/LLM systems, plus the validation discipline that makes any metric trustworthy.

Classification: Confusion Matrix & Derived Metrics

Confusion matrix is the source of truth: TP, FP, TN, FN. Every scalar classification metric is a function of these four counts (or their multiclass generalization).
Accuracy = (TP+TN)/total. Misleading under class imbalance: a model that always predicts the majority class scores 99% when the positive class is 1%.
Precision = TP/(TP+FP): of predicted positives, how many are correct? Optimize when false positives are costly (spam blocking, flagging cases for expensive human review).
Recall (Sensitivity, TPR) = TP/(TP+FN): of actual positives, how many did we catch? Optimize when misses are costly (cancer screening, fraud detection).
F1 = harmonic mean of precision and recall = 2PR/(P+R). Fβ weights recall β² times as much as precision (β>1 favors recall, β<1 favors precision). Use F1 for a single balanced score on imbalanced classes.
Specificity (TNR) = TN/(TN+FP); FPR = FP/(FP+TN) = 1 − specificity. Relevant for ROC analysis and false-alarm-sensitive settings.
Balanced accuracy = (TPR+TNR)/2 averages per-class recall, a robust single number under imbalance.
Multiclass averaging: macro (unweighted mean over classes, treats classes equally and surfaces rare-class failure), micro (pool counts globally; equals accuracy in single-label settings), weighted (mean weighted by class support).

Threshold-Free & Probabilistic Metrics

ROC-AUC: probability that a random positive is ranked above a random negative. Threshold-independent, but optimistic under heavy imbalance because the large TN count keeps FPR low.
PR-AUC (Average Precision): area under the precision-recall curve. Preferred over ROC-AUC when positives are rare; its no-skill baseline equals the positive prevalence, not 0.5.
Log loss (cross-entropy) penalizes confident wrong predictions sharply and rewards calibrated probabilities; it is the objective most probabilistic classifiers actually train on.
Brier score = mean squared error of predicted probabilities (for the positive class); decomposes into calibration + refinement. Lower is better.
Calibration ≠ discrimination: a model can have high AUC yet output miscalibrated probabilities. Diagnose with reliability diagrams / ECE; fix with Platt scaling (logistic) or isotonic regression.
Threshold selection is a business decision, not a metric: pick the operating point from the PR or ROC curve given the cost ratio of FP to FN, then report metrics at that threshold.

Regression Metrics

MAE = mean |error|: in target units, robust to outliers, and minimized by predicting the conditional median.
MSE / RMSE: square errors so large mistakes dominate; RMSE is in target units and is minimized by predicting the conditional mean. Use when big errors are disproportionately bad.
R² = 1 − SS_res/SS_tot: fraction of variance explained; can be negative (worse than predicting the mean). Use Adjusted R² to compare models with different feature counts (it penalizes added predictors).
MAPE = mean |error/actual|: scale-free and readable as a %, but explodes near zero and is asymmetric — it penalizes over-prediction more than under-prediction (so it biases models toward under-forecasting). Prefer sMAPE or WAPE for robustness.
Huber / quantile loss: Huber blends MAE and MSE for outlier robustness; quantile (pinball) loss targets specific percentiles to build prediction intervals.
Always pair an error metric with a naive baseline (mean, last value, seasonal naive) so the number has context.

Ranking, Recommendation & Retrieval

Precision@k / Recall@k: relevance within the top-k results, matching how users actually experience search and recsys.
MAP (Mean Average Precision): mean over queries of the average precision computed at each relevant hit; rewards ranking relevant items higher.
NDCG@k: discounted cumulative gain normalized by the ideal ranking (range 0-1); supports graded (not just binary) relevance and is the standard for search quality.
MRR (Mean Reciprocal Rank): mean of 1/rank of the first relevant item; ideal when only the top hit matters (QA, autocomplete).
Beyond accuracy: Hit Rate, Coverage, Diversity, and Novelty guard against popularity-bias collapse and filter bubbles.
For RAG retrieval: report context precision/recall and whether the gold passage appears in the retrieved set, separately from answer quality.

Generative & LLM Evaluation (2026-current)

Classic overlap metrics — BLEU (translation), ROUGE (summarization), METEOR — measure n-gram overlap; cheap but weak proxies for semantic quality.
Embedding-based metrics such as BERTScore capture meaning beyond surface tokens and correlate better with human judgment.
Perplexity measures language-model fit (lower = better) but says nothing about usefulness, factuality, or instruction-following.
LLM-as-judge is now standard: a strong model scores outputs against rubrics (helpfulness, faithfulness, safety). Mitigate position, verbosity, and self-preference bias; prefer pairwise comparison over absolute scores and calibrate against human labels.
RAG-specific frameworks (e.g., RAGAS-style) score faithfulness/groundedness, answer relevancy, and context relevancy to separate hallucination from retrieval failure.
Task-level evals beat single scores: pass@k for code, exact-match/F1 for QA, plus golden-set regression suites and human preference (win-rate / Elo / Bradley-Terry) for open-ended generation.
Treat safety and robustness evals (toxicity, jailbreak resistance, refusal correctness) as first-class metrics, not afterthoughts.

Validation Discipline & Statistical Rigor

A metric is only as trustworthy as its split: tune on validation/CV, then report final numbers exactly once on a held-out test set never touched during development.
Cross-validation: k-fold for i.i.d. data, StratifiedKFold to preserve class ratios, GroupKFold to prevent entity leakage, TimeSeriesSplit (forward chaining) for temporal data — never shuffle time series.
Report uncertainty: bootstrap confidence intervals and significance tests (paired t-test / McNemar for classifiers / Wilcoxon) before claiming model A beats model B.
Guard against leakage: target leakage, train/test contamination, and fitting scalers/encoders/imputers on the full dataset before splitting all inflate metrics — fit them inside the CV loop (use a Pipeline).
Offline ≠ online: confirm with A/B tests and guardrail metrics; an offline winner can lose on latency, fairness, or business KPIs.
Monitor in production: track metric decay, data/concept drift, and slice-level performance (fairness across subgroups) over time, not just an aggregate.

Common pitfalls

Reporting accuracy on imbalanced data — use PR-AUC, F1, balanced accuracy, or per-class metrics instead.
Using ROC-AUC as the headline metric for rare-positive problems where PR-AUC is far more honest.
Confusing precision and recall, or optimizing one without acknowledging the trade-off and the implied threshold.
Tuning hyperparameters on the test set (or peeking repeatedly), producing optimistic, non-reproducible results.
Data leakage: fitting scalers/encoders/imputers before the train-test split, or letting future info into time-series folds.
Treating high AUC as proof of good probabilities, ignoring calibration when decisions depend on the actual probability value.
Comparing models on a single run without confidence intervals or significance testing; small gaps are often noise.
Blindly trusting BLEU/ROUGE for LLMs, or trusting an LLM judge without checking it against human labels and debiasing it.
Reporting only aggregate metrics while a critical subgroup or slice silently fails (fairness/robustness blind spot).
Optimizing an offline metric that is misaligned with the actual business objective or online behavior.

Interview tips

Expect 'which metric and why for THIS problem' — anchor the answer to the cost of false positives vs false negatives and the class balance.
Be ready to derive precision, recall, F1, TPR, and FPR from a confusion matrix on the spot, and explain ROC vs PR curves.
Explain the precision-recall trade-off and how moving the decision threshold shifts each; tie threshold choice to business cost.
Articulate why ROC-AUC can mislead on imbalanced data and when you would switch to PR-AUC or per-class metrics.
Distinguish discrimination from calibration and name fixes (Platt scaling, isotonic regression); interviewers value this nuance.
For regression, justify MAE vs RMSE (outlier sensitivity, median vs mean) and explain negative R².
Know the cross-validation variants and which one prevents leakage for grouped, temporal, or imbalanced data.
For modern roles, discuss evaluating LLM/RAG systems: LLM-as-judge pitfalls, faithfulness vs relevancy, pass@k, and human win-rate.
Always mention validating offline metrics with online A/B tests and monitoring for drift in production.

Read the full lesson Test yourself — 40 question quiz