ROC and precision-recall curves

ROC stands for Receiver Operating Characteristic. It is a curve that plots the True Positive Rate (TPR) on the y-axis against the False Positive Rate (FPR) on the x-axis, tracing classifier performance across all possible decision thresholds from 0 to 1. The term originated in signal detection theory during World War II.

The y-axis plots the True Positive Rate (TPR = TP / (TP + FN)), also called sensitivity or recall. The x-axis plots the False Positive Rate (FPR = FP / (FP + TN)), also called fall-out. Every point on the curve corresponds to a different classification threshold, sweeping from predicting all negative to all positive.

A random classifier achieves an AUC of approximately 0.5. Its ROC curve is a straight diagonal line from (0, 0) to (1, 1), meaning it has no discriminatory power — it performs no better than flipping a fair coin to decide the class label.

TPR = TP / (TP + FN), where TP is true positives and FN is false negatives. It measures the proportion of all actual positive cases that the model correctly identifies. A TPR of 0.9 means the model catches 90% of real positives, missing only 10%. It is also known as sensitivity or recall.

FPR = FP / (FP + TN), where FP is false positives and TN is true negatives. It measures the proportion of actual negative cases that are incorrectly classified as positive. FPR is also written as 1 − Specificity. A low FPR is desirable because it means few negatives are falsely flagged.

AUC stands for Area Under the Curve (specifically, the ROC curve). It equals the probability that a randomly chosen positive example receives a higher predicted score than a randomly chosen negative example. An AUC of 1.0 is perfect discrimination; 0.5 is no better than random guessing.

Precision = TP / (TP + FP). It measures what fraction of all positive predictions are actually correct. High precision means the model rarely fires false alarms. For example, in spam detection, high precision means most emails flagged as spam are indeed spam, and few legitimate emails are wrongly blocked.

Recall = TP / (TP + FN), which is identical to the True Positive Rate. It measures what fraction of all actual positives are found by the model. Unlike precision, which penalises false positives, recall penalises false negatives. In cancer screening, high recall is critical so no disease is missed.

The point (0, 0) corresponds to a classification threshold so high that the model predicts every example as negative. This gives FPR = 0 (no false positives at all) but also TPR = 0 (no true positives either). No examples are positively classified, so the model is maximally conservative.

The point (1, 1) represents a threshold so low that the model predicts every example as positive. This gives TPR = 1 (all positives found) but FPR = 1 (all negatives are also wrongly called positive). Maximum recall is achieved, but specificity drops to zero — the model is maximally permissive.

Changing the threshold does not redraw the ROC curve. The full curve is generated by sweeping all possible thresholds and plotting the resulting (FPR, TPR) pairs. Changing the threshold in practice moves your operating point along that fixed curve — lowering the threshold increases both TPR and FPR, sliding the point toward (1, 1).

In highly imbalanced data (e.g., 0.1% positive class), the huge number of true negatives keeps FPR very small even if there are many false positives, making the ROC curve look deceptively good. The PR curve uses precision, which directly reflects false positives relative to predicted positives, exposing poor performance that the ROC curve conceals.

`roc_curve(y_true, y_scores)` returns three arrays: `fpr` (false positive rates), `tpr` (true positive rates), and `thresholds` (the decision thresholds at each point). You plot `fpr` on the x-axis and `tpr` on the y-axis to draw the ROC curve, and use `auc(fpr, tpr)` from the same library to compute the area under it.

from sklearn.metrics import roc_curve, auc
fpr, tpr, thresholds = roc_curve(y_true, y_scores)
roc_auc = auc(fpr, tpr)
print(f'AUC = {roc_auc:.3f}')

The F1 score is the harmonic mean of precision and recall: F1 = 2 × (P × R) / (P + R). It gives a single performance number at one chosen threshold, corresponding to a single point on the PR curve. The PR curve shows the full precision-recall trade-off across all thresholds; F1 summarises performance only at the selected operating threshold.

AUC-ROC equals the probability that, if you randomly pick one positive example and one negative example from the dataset, the model assigns a higher predicted score to the positive. For example, AUC = 0.85 means the model correctly ranks a random positive above a random negative 85% of the time. This makes AUC a threshold-independent ranking metric.

A perfect classifier's ROC curve forms an 'L-shape': it rises straight from (0, 0) to (0, 1) (achieving TPR = 1 with zero false positives) and then across to (1, 1). This curve hugs the top-left corner and has AUC = 1.0, meaning there exists a threshold that perfectly separates all positives from all negatives.

AUC-PR (also called Average Precision, AP) summarises the precision-recall trade-off as a weighted mean of precision values at each recall level, where the weight is the change in recall. It is most informative for imbalanced problems — such as fraud detection or rare disease diagnosis — where the positive class is scarce and you care more about performance on positives than on negatives.

Lowering the threshold means more examples are predicted positive. Recall increases because more true positives are captured (fewer positives are missed). However, precision typically decreases because the additional predictions include more false positives. This fundamental trade-off is exactly what the precision-recall curve visualises across all thresholds.

`average_precision_score` computes AP = Σ (R_n − R_{n−1}) × P_n, a step-function (right-rectangular) integral. The trapezoid rule would linearly interpolate between PR points, which can overestimate AUC-PR because precision can drop sharply between thresholds. The step approach gives a conservative, more accurate estimate of true PR area.

from sklearn.metrics import average_precision_score
ap = average_precision_score(y_true, y_scores)
print(f'Average Precision = {ap:.3f}')

The most common extension is One-vs-Rest (OvR): for each class, one ROC curve is drawn treating that class as positive and all others as negative, giving one AUC per class. These per-class AUCs are then averaged (macro: unweighted; weighted: proportional to class size). Scikit-learn's `roc_auc_score` supports both OvR and One-vs-One (OvO) strategies.

A better classifier's curve bows toward the top-left corner (low FPR, high TPR). The ideal point (0, 1) means every positive is found while no negative is falsely flagged. The further the curve bows toward the top-left, the larger the AUC, indicating stronger ability to separate positive from negative examples across all thresholds.

You compute fpr and tpr using `roc_curve`, then plot the classifier's curve and add a dashed diagonal (the random-classifier baseline at AUC = 0.5). The diagonal from (0,0) to (1,1) is the reference: any model whose curve lies above it is better than random.

import matplotlib.pyplot as plt
from sklearn.metrics import roc_curve, auc

fpr, tpr, _ = roc_curve(y_true, y_scores)
roc_auc = auc(fpr, tpr)

plt.figure(figsize=(6, 5))
plt.plot(fpr, tpr, label=f'Model (AUC = {roc_auc:.2f})')
plt.plot([0, 1], [0, 1], 'k--', label='Random (AUC = 0.50)')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
plt.legend()
plt.show()

TPR and FPR are both computed within their own class (TP+FN for positives, FP+TN for negatives), so changing the ratio of positives to negatives does not directly alter these rates. Precision = TP/(TP+FP) mixes both classes: when negatives are dominant, even a small FPR produces many false positives, sharply dropping precision and making AUC-PR sensitive to the class ratio.

Precision = P(actually positive | predicted positive). By Bayes' theorem: P(+|pred+) = P(pred+|+)·P(+) / P(pred+) = (TPR·π) / (TPR·π + FPR·(1−π)). This shows that even with high TPR and low FPR, precision can be very low when prevalence π is tiny — the core reason ROC looks optimistic while PR reveals true difficulty on rare events.

AUC-ROC is exactly equal to the normalised Wilcoxon-Mann-Whitney statistic. Both count the number of (positive, negative) sample pairs where the model scores the positive higher than the negative, divided by the total number of such pairs. This equivalence gives AUC its probabilistic interpretation and enables non-parametric significance tests (e.g., DeLong's test) when comparing two AUC values.

The concordance index (C-statistic) is numerically equal to AUC-ROC for binary classification. It measures the probability that, for a randomly chosen positive-negative pair, the model correctly assigns the higher score to the positive. A C-statistic of 0.5 indicates no discrimination; 1.0 is perfect. It is widely used in medicine (e.g., survival models) because of this direct probabilistic meaning.

When ROC curves cross, Model A may be superior at low FPR while Model B is superior at high FPR. A single AUC collapses this into one number and may declare a winner that is actually worse in the specific operating region you need. The correct approach is to compare Partial AUC (pAUC) restricted to the FPR range that is operationally relevant for your application.

Discrimination (AUC) asks whether the model correctly ranks positives above negatives — it is about ordering, not magnitude. Calibration asks whether predicted probabilities match actual frequencies: cases predicted at 70% should be positive about 70% of the time. A model can have high AUC (good ranking) but poor calibration (e.g., all scores compressed near 0.6), or be well-calibrated but have weak discrimination.

The Youden index J = TPR − FPR = Sensitivity + Specificity − 1. The optimal threshold maximises J, identifying the point on the ROC curve that is farthest from the diagonal. In numpy/sklearn, after calling `roc_curve`, you find the index of the maximum (tpr − fpr) and retrieve the corresponding threshold value.

import numpy as np
from sklearn.metrics import roc_curve

fpr, tpr, thresholds = roc_curve(y_true, y_scores)
youden_index = tpr - fpr
optimal_idx = np.argmax(youden_index)
optimal_threshold = thresholds[optimal_idx]
print(f'Optimal threshold: {optimal_threshold:.3f}')
print(f'TPR: {tpr[optimal_idx]:.3f}, FPR: {fpr[optimal_idx]:.3f}')

AUC-ROC integrates performance across all thresholds, many of which will never be used in production. With 0.05% fraud prevalence, even a tiny FPR floods the system with false alarms. By the Bayes formula, precision = (TPR·π)/(TPR·π + FPR·(1−π)) can be extremely low. Analysts must investigate every alert; low precision means enormous wasted effort, making the model impractical despite its high AUC.

Macro-averaged AUC computes AUC separately for each class (OvR) and takes their unweighted mean, giving every class equal importance regardless of frequency. Micro-averaged AUC pools all one-vs-rest prediction scores and labels together into a single combined ROC curve and computes AUC on that, naturally weighting by class size so the majority class dominates. Use macro when classes matter equally; use micro when overall population-level performance is the goal.

An AUC below 0.5 means the model is performing worse than random guessing — it systematically assigns higher scores to negatives than to positives. Crucially, simply flipping all predictions (predict positive when the model says negative) would produce a model with AUC > 0.5. This usually signals a label-encoding bug, a data leakage issue, or a fundamental flaw in feature construction.

Unlike the ROC curve, precision on a PR curve can drop sharply (non-monotonically) between adjacent thresholds when a false positive is added. Linear interpolation between those points overestimates the area under the curve. Sklearn's `average_precision_score` uses step (right-rectangular) interpolation — it takes precision at the right endpoint of each recall interval — giving a conservative, more accurate estimate of true PR area.