Model evaluation metrics

Accuracy is the fraction of predictions the model got right: (correct predictions) / (total predictions). Using confusion-matrix terms it equals (TP + TN) / (TP + TN + FP + FN). It is intuitive but can be misleading when classes are imbalanced.

A confusion matrix is a 2x2 table comparing predicted vs actual labels. Its cells are True Positives (correctly predicted positive), True Negatives (correctly predicted negative), False Positives (negatives wrongly predicted positive), and False Negatives (positives wrongly predicted negative). It is the foundation for computing precision, recall, and accuracy.

Precision answers: of all the items the model labeled positive, what fraction were actually positive? Its formula is TP / (TP + FP). High precision means few false alarms.

Recall answers: of all the actual positives, what fraction did the model correctly catch? Its formula is TP / (TP + FN). High recall means few missed positives.

The F1 score is the harmonic mean of precision and recall: F1 = 2 * (precision * recall) / (precision + recall). It gives a single balanced number that is high only when both precision and recall are high, making it useful for imbalanced datasets where accuracy is misleading.

If 99% of samples belong to one class, a model that always predicts that majority class achieves 99% accuracy while learning nothing about the rare class. Accuracy hides this failure, so metrics like precision, recall, F1, or balanced accuracy are preferred for imbalanced data.

Specificity (true negative rate) measures how well the model identifies actual negatives: TN / (TN + FP). It is the counterpart to recall (sensitivity), which focuses on positives. High specificity means few false positives among the true negatives.

You would usually prioritize recall, because missing a sick patient (a false negative) is far more costly than a false alarm. High recall ensures most true cases are caught, even at the expense of some unnecessary follow-up tests.

Marking a legitimate email as spam (a false positive) can make a user miss something important, so you want high precision to minimize those mistakes. It is more acceptable to let a few spam emails through (lower recall) than to lose real messages.

An ROC (Receiver Operating Characteristic) curve plots the True Positive Rate (recall) on the y-axis against the False Positive Rate (1 - specificity) on the x-axis as the classification threshold is varied. It shows the trade-off between catching positives and raising false alarms across all thresholds.

AUC summarizes the ROC curve as a single number representing the probability that the model ranks a random positive higher than a random negative. AUC = 1.0 is a perfect classifier, while AUC = 0.5 indicates random guessing.

A Type I error is a false positive: the model predicts the positive class when the truth is negative. For example, flagging a healthy patient as sick. It corresponds to the FP cell of the confusion matrix.

A Type II error is a false negative: the model predicts the negative class when the truth is positive. For example, telling a sick patient they are healthy. It corresponds to the FN cell of the confusion matrix.

MAE is the average of the absolute differences between predicted and actual values: MAE = (1/n) * sum(|y_pred - y_true|). It is in the same units as the target and treats all errors linearly, so it is less sensitive to outliers than squared-error metrics.

MSE is the average of squared differences: MSE = (1/n) * sum((y_pred - y_true)^2). Because errors are squared, large errors contribute disproportionately more than small ones, making MSE very sensitive to outliers.

RMSE is the square root of MSE: RMSE = sqrt(MSE). Taking the square root brings the metric back to the same units as the target variable, making it easier to interpret while still penalizing large errors more than small ones.

R-squared measures the proportion of variance in the target that the model explains, ranging typically from 0 to 1 (it can be negative for very poor models). An R-squared of 0.8 means the model explains 80% of the variation; 1.0 means perfect fit and 0 means it does no better than predicting the mean.

Log loss measures the quality of probabilistic predictions by penalizing confident wrong answers heavily; for binary cases it is -(1/n) * sum(y*log(p) + (1-y)*log(1-p)). It requires the model to output probabilities rather than just hard labels, and lower log loss is better.

Precision = TP / (TP + FP) = 40 / (40 + 10) = 40/50 = 0.8, or 80%. This means 80% of the items predicted positive were truly positive.

Recall = TP / (TP + FN) = 40 / (40 + 20) = 40/60 ≈ 0.67, or about 67%. This means the model caught about 67% of all actual positives.

Macro-averaging computes the metric separately for each class and then takes the unweighted mean, treating all classes equally regardless of size. Micro-averaging pools all classes' TP, FP, FN together before computing the metric, so it is dominated by the larger classes.

Raising the threshold makes the model more conservative about predicting positive, typically increasing precision but lowering recall; lowering the threshold does the opposite. This precision-recall trade-off lets you tune the model to the costs of false positives versus false negatives.

Balanced accuracy is the average of recall computed on each class (for binary, the mean of sensitivity and specificity). It is preferred for imbalanced datasets because it gives equal weight to each class, preventing a majority-class bias from inflating the score.

Performance on the training set can be overly optimistic because the model may have memorized those examples (overfitting). Evaluating on a held-out test set estimates how well the model generalizes to new, unseen data, which is what we actually care about.

If a model scores very high on the training set but much lower on the test set, the large gap signals overfitting: it learned noise specific to the training data rather than general patterns. A small gap with good test performance indicates better generalization.

K-fold cross-validation splits the data into k equal parts, trains on k-1 folds and tests on the remaining fold, repeating so each fold serves as the test set once. Averaging the k scores gives a more reliable, less luck-dependent estimate of performance than a single train/test split.

A baseline gives a reference score that any useful model should beat; for example, on imbalanced data a majority-class predictor may already reach high accuracy. Comparing your model to the baseline shows whether it actually learned something meaningful rather than just exploiting class frequencies.

The F-beta score is a weighted harmonic mean of precision and recall where beta controls their relative importance. Beta > 1 weights recall more (e.g., F2), beta < 1 weights precision more (e.g., F0.5), and beta = 1 gives the balanced F1 score.

MAPE is the average of absolute errors expressed as a percentage of the actual values: (100/n) * sum(|y_true - y_pred| / |y_true|). It is easy to interpret as a percentage, but it becomes undefined or explodes when actual values are zero or near zero.

The precision-recall curve plots precision against recall across thresholds, while ROC plots TPR against FPR. PR curves are more informative for highly imbalanced datasets because they focus on the positive (rare) class and are not flattered by the large number of true negatives that can make ROC look optimistic.

Sensitivity and recall are two names for the same quantity, both equal to TP / (TP + FN), the true positive rate. The term 'sensitivity' is common in medicine while 'recall' is common in machine learning.

Ordinary R-squared never decreases when you add more predictors, even useless ones, which can be misleading. Adjusted R-squared penalizes the inclusion of extra features that do not improve the model, so it only increases when a new predictor genuinely helps.

sklearn.metrics.classification_report(y_true, y_pred) returns a formatted report with precision, recall, F1-score, and support for each class, plus averaged summaries. It is a quick way to inspect per-class performance.

from sklearn.metrics import classification_report
print(classification_report(y_true, y_pred))

Top-k accuracy counts a prediction as correct if the true label is among the model's k highest-probability predictions. It is common in multi-class problems with many classes, such as image classification (e.g., top-5 accuracy on ImageNet), where being 'close' is still useful.

The silhouette score evaluates clustering quality by measuring how similar each point is to its own cluster versus the nearest other cluster. It ranges from -1 to 1, where values near 1 indicate well-separated, cohesive clusters and negative values suggest points may be in the wrong cluster.

sklearn.metrics.confusion_matrix(y_true, y_pred) returns a 2D array where rows correspond to actual classes and columns to predicted classes. Each entry counts how many samples of the true class (row) were predicted as the column class, with the diagonal holding correct predictions.

from sklearn.metrics import confusion_matrix
print(confusion_matrix(y_true, y_pred))

Accuracy ignores how errors are distributed between false positives and false negatives. Two models can share an accuracy value yet have very different precision/recall trade-offs, so one may suit a cost-sensitive task far better than the other depending on which error type is worse.

Cohen's kappa measures agreement between predictions and actual labels while correcting for the agreement expected by random chance. Unlike raw accuracy, a kappa near 0 means the model is no better than guessing, and values near 1 indicate strong agreement beyond chance.

A classifier is well-calibrated when its predicted probabilities match observed frequencies: among samples it assigns a 0.7 probability, roughly 70% should actually be positive. Calibration matters when you use the probabilities themselves for decisions, not just the final class label.

F1 = 2 * (0.6 * 0.4) / (0.6 + 0.4) = 2 * 0.24 / 1.0 = 0.48. The harmonic mean pulls the result below the arithmetic mean (0.5) because precision and recall are unequal.