Confusion matrix and precision/recall

A confusion matrix is a table that summarizes a classifier's predictions versus the actual labels. For binary classification it is a 2x2 table whose four cells are True Positives, False Positives, True Negatives, and False Negatives. It lets you see not just how many predictions were right, but exactly what kinds of mistakes the model made.

A True Positive is a case where the actual label is positive and the model also predicted positive. In other words, the model correctly identified a positive instance. For a disease test, a TP is a sick patient correctly flagged as sick.

A False Positive occurs when the model predicts positive but the actual label is negative, so it raises a false alarm. It is called a Type I error in statistics because it corresponds to wrongly rejecting a true null hypothesis (declaring an effect that is not there). An example is flagging a healthy patient as sick.

A False Negative happens when the model predicts negative but the true label is positive, meaning it misses a real positive case. In medical screening this is dangerous because a sick patient is told they are healthy and may not receive treatment. It is also known as a Type II error.

Accuracy is the fraction of all predictions that are correct: Accuracy = (TP + TN) / (TP + TN + FP + FN). It treats every correct prediction equally regardless of class. Although intuitive, it can be misleading when the classes are imbalanced.

Precision measures how many of the instances predicted as positive are actually positive: Precision = TP / (TP + FP). It answers the question, 'When the model says positive, how often is it right?' High precision means few false alarms.

Recall, also called sensitivity or the true positive rate, measures how many of the actual positives the model successfully found: Recall = TP / (TP + FN). It answers, 'Of all the real positive cases, how many did we catch?' High recall means few missed positives.

Precision = TP / (TP + FP) = 80 / (80 + 20) = 0.80, or 80%. Recall = TP / (TP + FN) = 80 / (80 + 40) = 0.667, or about 66.7%. So the model is right 80% of the time when it predicts positive but only catches two-thirds of all real positives.

When one class dominates, a model can score high accuracy simply by always predicting the majority class while completely failing on the minority class. For example, if 99% of emails are not spam, a model that labels everything 'not spam' is 99% accurate but catches zero spam. Precision and recall reveal this failure that accuracy hides.

The F1 score is the harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). It gives a single number that balances both, and being a harmonic mean it stays low if either precision or recall is low. It is especially useful for imbalanced datasets where accuracy is unhelpful.

Using F1 = 2 * (P * R) / (P + R) = 2 * (0.6 * 0.6) / (0.6 + 0.6) = 2 * 0.36 / 1.2 = 0.6. When precision and recall are equal, the F1 score equals that same value. This is because the harmonic mean of two equal numbers is the number itself.

The harmonic mean is dominated by the smaller of the two values, so F1 is only high when both precision and recall are high. An arithmetic mean could give a misleadingly good score even if one metric is near zero (e.g. precision 1.0 and recall 0.0 average to 0.5). The harmonic mean penalizes such imbalance, giving F1 = 0 in that extreme case.

A False Positive (a legitimate email wrongly marked as spam) is usually considered worse because the user may miss an important message hidden in the spam folder. A False Negative (a spam email reaching the inbox) is merely annoying. Therefore spam filters are typically tuned to favor high precision on the spam class.

You should prioritize recall, because missing a real cancer case (a False Negative) can be fatal, so you want to catch as many true positives as possible. A False Positive only leads to further testing, which is far less dangerous than a missed diagnosis. Hence screening tests are tuned for high sensitivity (recall).

Most classifiers output a probability, and we apply a threshold to decide the predicted class. Raising the threshold makes the model more cautious about predicting positive, which typically increases precision but lowers recall; lowering it does the opposite. You choose the threshold based on whether false alarms or missed cases are more costly.

Specificity, the true negative rate, measures how many of the actual negatives are correctly identified: Specificity = TN / (TN + FP). It answers, 'Of all the real negative cases, how many did we correctly label negative?' It is the negative-class counterpart of recall.

If everything is predicted positive, there are no False Negatives, so recall = TP / (TP + 0) = 1.0 (100%). Precision, however, becomes TP / (TP + FP), which is just the proportion of positives in the data and is usually low when negatives are common. This shows recall alone can be trivially maximized.

You can use sklearn.metrics. confusion_matrix returns the 2x2 table and precision_score computes TP/(TP+FP). For y_true=[1,0,1,1,0] and y_pred=[1,0,0,1,1], TP=2, FP=1, FN=1, TN=1, giving precision = 2/3 = 0.667. The code below prints both.

from sklearn.metrics import confusion_matrix, precision_score
y_true = [1, 0, 1, 1, 0]
y_pred = [1, 0, 0, 1, 1]
print(confusion_matrix(y_true, y_pred))
print(precision_score(y_true, y_pred))  # 0.667

Sensitivity (recall) measures the proportion of actual positives correctly identified, TP/(TP+FN), while specificity measures the proportion of actual negatives correctly identified, TN/(TN+FP). Sensitivity focuses on not missing positives; specificity focuses on not falsely flagging negatives. A good test balances both depending on the application.

By default sklearn arranges rows by the true class and columns by the predicted class, in ascending label order. For binary labels [0,1], the matrix is [[TN, FP], [FN, TP]]: top-left is TN, top-right is FP, bottom-left is FN, bottom-right is TP. Knowing this layout is essential to read the output correctly.

The False Positive Rate is the fraction of actual negatives that are incorrectly classified as positive: FPR = FP / (FP + TN). It equals 1 minus specificity. FPR is plotted on the x-axis of an ROC curve.

TP = 150 (correctly flagged frauds), FP = 200 - 150 = 50 (false alarms), FN = 50 (missed frauds). Precision = 150 / (150 + 50) = 0.75, and recall = 150 / (150 + 50) = 0.75. So the model is 75% reliable when it flags fraud and catches 75% of all fraud.

For K classes the confusion matrix becomes a K x K table where entry (i, j) counts samples whose true class is i but were predicted as class j. The diagonal holds correct predictions and off-diagonal entries hold the misclassifications. Per-class precision and recall are then computed by treating each class as 'positive' in a one-vs-rest manner.

Macro-averaging computes precision for each class separately and then takes the unweighted mean, treating all classes equally regardless of size. Micro-averaging pools all the TP and FP counts across classes first and then computes a single precision, so larger classes dominate. Macro is preferred when you care about rare classes; micro reflects overall instance-level performance.

High precision with low recall means that the few predictions the model labels positive are almost always correct, but it misses the vast majority of actual positives. The model is overly conservative, predicting positive only when extremely confident. Its F1 score would be low, around 0.18, signaling poor overall positive-class coverage.

The diagonal entries of a confusion matrix represent the correctly classified samples, where the predicted class matches the true class. In a 2x2 matrix these are the TN and TP cells. The sum of the diagonal divided by the total number of samples equals the accuracy.

Accuracy only counts total correct predictions and ignores how errors are split between False Positives and False Negatives. Two models can make the same number of mistakes overall but distribute them differently: one may favor false alarms (lower precision) while the other favors missed cases (lower recall). Looking at the confusion matrix reveals these differences that accuracy hides.

The F-beta score generalizes F1 by weighting recall beta-squared times more than precision. When beta > 1 (e.g. F2), recall is weighted more heavily, which is useful when missing positives is costlier than false alarms. When beta < 1, precision is emphasized instead.

sklearn.metrics.classification_report prints per-class precision, recall, F1-score, and support in one summary, plus macro and weighted averages. It is the quickest way to inspect all the key metrics at once. The code below shows its use on small label lists.

from sklearn.metrics import classification_report
y_true = [1, 0, 1, 1, 0, 0]
y_pred = [1, 0, 0, 1, 0, 1]
print(classification_report(y_true, y_pred))

An ROC curve plots the True Positive Rate (recall) against the False Positive Rate across thresholds, while a precision-recall curve plots precision against recall. The PR curve is often more informative than ROC for highly imbalanced data because it focuses on the positive class and is not dominated by the large number of true negatives. Both summarize threshold trade-offs in a single plot.

Accuracy = (TP + TN) / total = (50 + 900) / 1000 = 950 / 1000 = 0.95, or 95%. However, with only 50 of the 90 actual positives caught, recall is just about 55.6%, so the high accuracy hides weak positive-class performance. This illustrates why you should not rely on accuracy alone.

Support is the number of actual occurrences of each class in the true labels, i.e. how many samples truly belong to that class. It tells you how much data backs each precision and recall figure, so a metric with very low support may be unreliable. Support does not change with the model's predictions, only with the dataset.

Yes. Precision = 1.0 means there are no False Positives and recall = 1.0 means there are no False Negatives, so both being 1.0 implies FP = 0 and FN = 0. This corresponds to a perfect classifier that makes no errors on the positive class. In practice this is rare and may signal data leakage or overfitting if seen on real data.

Negative predictive value is the precision of the negative class: NPV = TN / (TN + FN), the fraction of negative predictions that are truly negative. While precision tells you how trustworthy a positive prediction is, NPV tells you how trustworthy a negative prediction is. Both together give a fuller picture, which matters in screening where a reassuring 'negative' result must be reliable.