ROC curves and AUC

ROC stands for Receiver Operating Characteristic. The name originates from World War II radar signal detection, where operators measured how well they could distinguish enemy signals from noise. In machine learning, an ROC curve shows how a binary classifier's performance changes as the decision threshold varies.

An ROC curve plots the True Positive Rate (TPR, also called recall or sensitivity) on the y-axis against the False Positive Rate (FPR) on the x-axis. Each point on the curve corresponds to a particular classification threshold. As the threshold changes, both rates change, tracing out the curve.

TPR equals TP / (TP + FN), where TP is the number of true positives and FN is the number of false negatives. It measures the fraction of actual positive cases that the model correctly identifies. TPR is also known as sensitivity or recall.

FPR equals FP / (FP + TN), where FP is the number of false positives and TN is the number of true negatives. It measures the fraction of actual negative cases that the model incorrectly labels as positive. FPR is equal to 1 minus specificity.

AUC stands for Area Under the (ROC) Curve. It is a single number summarizing the entire ROC curve, ranging from 0 to 1. A perfect classifier has AUC = 1, while a random classifier has AUC = 0.5.

A random classifier has an AUC of 0.5, which corresponds to the diagonal line from the bottom-left to the top-right of the ROC plot. This represents no discriminative ability between the two classes. Any useful model should achieve an AUC above 0.5.

The diagonal line represents the performance of a random (no-skill) classifier, where TPR always equals FPR. A model whose ROC curve lies on this line has an AUC of 0.5 and provides no useful discrimination. Good classifiers produce curves that bow toward the top-left corner, above this line.

The top-left corner, at coordinates (0, 1), represents the ideal classifier. At this point the False Positive Rate is 0 and the True Positive Rate is 1, meaning every positive is caught and no negatives are misclassified. The closer an ROC curve approaches this corner, the better the model.

The classifier outputs a probability score for each instance, and the decision threshold is varied from high to low. At each threshold, predictions are made and the resulting TPR and FPR are computed, producing one point. Connecting all these points across thresholds traces out the ROC curve.

AUC equals the probability that a randomly chosen positive example is ranked higher (given a higher score) by the model than a randomly chosen negative example. This means AUC measures how well the model orders or ranks positives above negatives. An AUC of 0.9, for instance, means there is a 90% chance a random positive outranks a random negative.

AUC summarizes performance across all possible decision thresholds rather than just one fixed cut-off, so it does not depend on choosing a specific threshold. Accuracy, by contrast, is computed at a single threshold and can change dramatically if that threshold shifts. This makes AUC a more robust measure of a model's overall ranking ability.

On imbalanced datasets, accuracy can be misleadingly high because a model can score well by simply predicting the majority class. AUC instead evaluates how well the model ranks positives above negatives, which is less affected by the class proportions. This makes AUC a more informative metric when one class is rare.

Specificity equals TN / (TN + FP), the fraction of actual negatives correctly identified. The False Positive Rate is exactly 1 minus specificity. So as specificity increases, FPR decreases, and the ROC x-axis (FPR) can be thought of as showing 1 minus specificity.

As the threshold approaches zero, the model labels almost everything as positive, so both TPR and FPR approach 1. This corresponds to the top-right corner (1,1) of the ROC plot. Conversely, raising the threshold toward one drives both rates toward the origin (0,0).

The function roc_auc_score(y_true, y_score) from sklearn.metrics computes the AUC directly. It takes the true binary labels and the predicted probabilities (or scores) and returns a single AUC value. Separately, roc_curve can be used to get the FPR, TPR, and threshold points for plotting.

from sklearn.metrics import roc_auc_score
auc = roc_auc_score(y_true, y_scores)
print(auc)

roc_curve(y_true, y_score) returns three arrays: false positive rates (fpr), true positive rates (tpr), and the thresholds used. These can be plotted with fpr on the x-axis and tpr on the y-axis to draw the ROC curve. The thresholds array indicates the score cut-off corresponding to each point.

from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(y_true, y_scores)

Model A, with the higher AUC of 0.92, has better overall discriminative ability, meaning it ranks positives above negatives more reliably. A higher AUC indicates the model separates the two classes better across all thresholds. However, the best threshold choice still depends on the specific application's costs.

An AUC below 0.5 means the classifier performs worse than random guessing, systematically ranking negatives above positives. Interestingly, this often signals that the predictions are inverted, and flipping them (relabeling positive as negative) would yield an AUC above 0.5. It usually points to a bug or a label-ordering mistake.

The area is approximated using the trapezoidal rule, which sums the areas of trapezoids formed between consecutive (FPR, TPR) points. For each adjacent pair of points, the trapezoid area is the average height times the width in FPR. Adding these areas gives the total AUC. Scikit-learn's sklearn.metrics.auc(fpr, tpr) does exactly this.

from sklearn.metrics import auc
area = auc(fpr, tpr)  # trapezoidal integration of the ROC points

A perfect classifier's ROC curve rises straight up from (0,0) to (0,1) and then goes straight across to (1,1), hugging the top-left corner. This shape gives an AUC of exactly 1.0. It means there exists a threshold at which TPR is 1 and FPR is 0 simultaneously.

An ROC curve plots TPR against FPR, while a Precision-Recall (PR) curve plots precision against recall. PR curves focus on the positive class and are more informative when classes are highly imbalanced, because precision is sensitive to false positives relative to true positives. ROC curves can look overly optimistic under severe imbalance.

The ROC AUC is largely insensitive to class balance because both TPR and FPR are computed as rates within their own class (positives and negatives separately). Changing the ratio of positives to negatives does not directly change these per-class rates. This is a key reason ROC AUC is popular, though Precision-Recall metrics are often more revealing for rare-positive problems.

Youden's J statistic is defined as J = TPR - FPR (equivalently sensitivity + specificity - 1) at a given threshold. The threshold that maximizes J corresponds to the ROC point farthest above the diagonal, often chosen as an optimal operating point. It balances catching positives against avoiding false alarms.

The threshold should be lowered so the model classifies more cases as positive, increasing the True Positive Rate (sensitivity) and reducing missed cases (false negatives). This comes at the cost of a higher False Positive Rate, meaning more false alarms. The choice moves the operating point toward the upper-right region of the ROC curve.

ROC AUC is extended to multi-class problems using strategies like One-vs-Rest (OvR), where each class is treated as positive against all others, or One-vs-One (OvO), comparing class pairs. The per-class AUCs are then averaged, often using macro-averaging (unweighted) or weighted averaging by class frequency. Scikit-learn supports this via the multi_class parameter in roc_auc_score.

roc_auc_score(y_true, y_proba, multi_class='ovr')

When ROC curves cross, one model is better in some threshold ranges (e.g., low FPR) while the other is better elsewhere. AUC, being a single aggregate number, can hide this trade-off, so two models can have similar AUC yet behave very differently at the threshold you care about. In such cases you should compare performance in the specific operating region relevant to your application.

If every instance gets the same score, the model cannot distinguish classes, so its ROC curve is the diagonal line and its AUC is 0.5. There is effectively only one meaningful threshold behavior, giving no separation between positives and negatives. This is equivalent to random guessing.

Each threshold converts the model's probability scores into hard class predictions, which produces a specific confusion matrix of TP, FP, TN, and FN counts. From that matrix you compute one TPR and one FPR value, giving a single point on the curve. As the threshold sweeps through all values, you get a different confusion matrix and point for each.

The Gini coefficient is related to AUC by the formula Gini = 2 * AUC - 1. It rescales AUC so that a random model gives Gini = 0 and a perfect model gives Gini = 1. This metric is common in credit scoring and finance for measuring discriminative power.

ROC curves require continuous scores so that the threshold can be swept across a range of values, producing many (FPR, TPR) points. If you pass hard 0/1 predictions, there is effectively only one threshold, yielding just a single interior point and a crude, uninformative curve. Probabilities (or decision-function scores) let the curve capture the full trade-off.

y_scores = model.predict_proba(X_test)[:, 1]
fpr, tpr, _ = roc_curve(y_test, y_scores)

Specificity equals 1 minus FPR, so specificity = 1 - 0.2 = 0.8. This means 80% of the actual negative cases are correctly classified as negative. At this operating point the model balances catching 80% of positives while correctly rejecting 80% of negatives.

AUC measures average ranking ability across all thresholds, but a real deployment uses one specific threshold where the relevant costs of false positives and false negatives apply. A model with high overall AUC might still perform badly in the narrow operating region you actually need, such as very low FPR. Therefore the chosen operating point and class-specific costs matter as much as the aggregate AUC.

AUC is mathematically equivalent to the normalized Mann-Whitney U statistic, which counts how often positive instances are ranked above negative instances across all positive-negative pairs. Dividing that count of correctly ordered pairs by the total number of pairs gives the AUC. This is why AUC can be interpreted as a probability of correct ranking.

Shuffling destroys any association between scores and labels, so the model effectively ranks positives and negatives at random. The expected AUC becomes approximately 0.5, the value for a no-skill classifier. This is a useful sanity check: a properly working model should score well above 0.5, and shuffling its predictions should collapse the AUC back to about 0.5.