Model calibration

A calibrated model is one whose predicted probabilities match real-world frequencies. If a model assigns a probability of 0.8 to 100 different events, roughly 80 of them should actually occur. Calibration lets you trust the model's probability output as a meaningful number for decision-making, not just a ranking score.

A reliability diagram plots the predicted probability (x-axis) against the actual fraction of positive outcomes (y-axis) across binned prediction ranges. The diagonal line y = x represents perfect calibration, where predicted confidence exactly matches observed accuracy. Points above the diagonal indicate underconfidence; points below indicate overconfidence.

An overconfident model is 'too sure' of its predictions — it assigns probabilities close to 0 or 1 more than the data warrants. On the calibration curve, this appears as points falling below the diagonal for high probability ranges. For example, if the model predicts P=0.9 for 100 examples but only 65 are actually positive, those predictions are overconfident.

The Brier Score is BS = (1/n) * Σ(p_i − y_i)², the mean squared error between predicted probabilities p_i and actual binary outcomes y_i (0 or 1). A score of 0 is perfect; the worst possible score is 1. It rewards models that are both confident and correct, penalizing overconfident wrong predictions.

from sklearn.metrics import brier_score_loss
y_true = [1, 0, 1, 1, 0]
y_prob = [0.9, 0.1, 0.8, 0.7, 0.3]
print(brier_score_loss(y_true, y_prob))  # lower is better

Log-loss = −[y·log(p) + (1−y)·log(1−p)] heavily penalizes predictions that are both confident and wrong — assigning a near-zero probability to the true class gives an enormous penalty. It is a proper scoring rule, meaning it is minimized in expectation only when predicted probabilities exactly equal true class probabilities. This directly incentivizes a model to output well-calibrated probabilities.

Platt scaling fits a logistic regression — with just two parameters A and B — on the model's raw output scores using a held-out validation set: P(y=1|score) = σ(A·score + B). This sigmoid transformation maps arbitrary scores (positive or negative) to well-calibrated probabilities in [0,1]. It was originally developed for SVMs but works for any model that outputs a score.

from sklearn.calibration import CalibratedClassifierCV
from sklearn.svm import SVC

# Platt scaling via sigmoid method
model = CalibratedClassifierCV(SVC(), method='sigmoid', cv=5)
model.fit(X_train, y_train)
probs = model.predict_proba(X_test)[:, 1]

Isotonic regression fits a non-decreasing piecewise constant function that maps uncalibrated scores to probabilities, with no assumed shape. It is more flexible than Platt scaling (which assumes a sigmoid) but requires a larger calibration set — typically thousands of samples — to avoid overfitting. Prefer isotonic regression when you have abundant data and suspect the calibration mapping is not sigmoid-shaped.

ECE partitions all predictions into M equal-width confidence bins and computes the weighted average of the gap between bin accuracy and bin confidence: ECE = Σ_m (|B_m|/n) × |acc(B_m) − conf(B_m)|. A lower ECE means better calibration. Typically M = 10 or 15 bins are used, and the metric gives more weight to bins containing more predictions.

Naive Bayes assumes all features are conditionally independent given the class — a strong assumption that is rarely true in real data. When features are actually correlated, the model treats them as independent sources of evidence and multiplies their effects, pushing predicted probabilities to extremes near 0 or 1. This makes Naive Bayes systematically overconfident, and post-hoc calibration with Platt scaling can dramatically improve its probability quality.

Gradient boosting adds trees to correct residuals greedily, which optimizes a loss focused on improving ranking and reducing error rather than producing accurate probability estimates. This greedy process tends to push predictions toward extreme values. The resulting outputs are typically overconfident and benefit significantly from post-hoc calibration methods like Platt scaling or isotonic regression.

CalibratedClassifierCV wraps any base classifier and applies either Platt scaling (method='sigmoid') or isotonic regression (method='isotonic') to produce calibrated probabilities. With cross-validation (cv parameter), it trains the base model on k−1 folds and calibrates on the held-out fold, preventing data leakage. The result is a model with a predict_proba method that returns calibrated probability estimates.

from sklearn.calibration import CalibratedClassifierCV
from sklearn.ensemble import GradientBoostingClassifier

base = GradientBoostingClassifier()
cal_model = CalibratedClassifierCV(base, method='isotonic', cv=5)
cal_model.fit(X_train, y_train)
probs = cal_model.predict_proba(X_test)[:, 1]

Temperature scaling divides every output logit z_i by a scalar T before applying softmax: p_i = softmax(z/T). When T > 1, the output distribution becomes flatter and less confident; when T < 1, it becomes peakier and more confident. T is learned after training by minimizing log-loss on a held-out validation set — a single-parameter fix that improves calibration without changing the model's accuracy.

import torch
import torch.nn.functional as F

logits = torch.tensor([2.0, 1.0, 0.5])  # raw network output
T = 2.0  # T > 1 → softer, less confident
calibrated_probs = F.softmax(logits / T, dim=0)
print(calibrated_probs)

Discrimination (measured by AUC/ROC) captures how well a model ranks positive examples above negative ones — it is order-sensitive and does not require accurate probabilities. Calibration measures whether the predicted probability magnitudes match actual outcome frequencies. A model with AUC = 0.95 can still be badly miscalibrated (e.g., confidently predicting 0.99 when the true probability is 0.7), and these two properties must be evaluated separately.

An underconfident model hedges its predictions too much, clustering predicted probabilities near 0.5 even when the true probability is very high or very low. On the calibration curve, this appears as points lying above the diagonal for moderate-to-high predicted probabilities — the model says P=0.6 but the true positive rate in that bin is 0.85. The model's predictions are too conservative to be actionable.

MCE = max_m |acc(B_m) − conf(B_m)| is the worst-case calibration gap across all confidence bins, while ECE is the sample-weighted average across bins. MCE is more appropriate for safety-critical applications (medicine, autonomous vehicles) where even one region of severe miscalibration is unacceptable. ECE can hide large errors in low-frequency bins by averaging them away.

With severe class imbalance (e.g., only 1% positives), models tend to output very low probabilities for nearly all examples to minimize training loss, making them systematically underestimate the minority class probability. The calibration curve is skewed: the model may never predict confidently in the positive class range. Techniques such as class-weighting, resampling, or post-hoc recalibration on a representative validation set help correct this bias.

Sharpness refers to how extreme (close to 0 or 1) a model's predicted probabilities are. High sharpness means the model makes strong, confident predictions. Sharpness is desirable — a vague model that always predicts 0.5 is useless — but only when combined with good calibration. An overconfident model can be sharp but miscalibrated; the ideal model is both sharp (confident) and calibrated (honest about its confidence).

A scoring rule S(p, y) is proper if it is minimized in expectation only when the predicted probability p equals the true probability — the rule rewards honesty about uncertainty. Log-loss (cross-entropy) and the Brier Score are both proper scoring rules. Accuracy (0/1 loss) is not proper, because it can be gamed by always predicting 0 or 1 without caring about true probabilities, so it does not incentivize calibration.

Label smoothing replaces hard targets (0 or 1) with soft values such as ε/K and 1 − ε + ε/K (e.g., 0.05 and 0.95 for binary). This prevents the model from learning to push logits to ±∞ to perfectly minimize cross-entropy on hard labels. By discouraging extreme logit values, label smoothing acts as a calibration regularizer that reduces overconfidence in the learned probability distributions.

Guo et al. (2017) showed that large, deep networks trained with batch normalization and weight decay achieve high accuracy but become increasingly overconfident. These models have sufficient capacity to memorize training labels but their softmax outputs do not reflect true uncertainty — the gap between accuracy and calibration grows with network depth and training duration. This is why temperature scaling is commonly applied as a post-hoc fix.

Histogram binning divides the [0,1] interval into M bins and replaces each bin's predicted probabilities with the observed fraction of positive outcomes in that bin on a held-out validation set. It is a simple non-parametric post-hoc calibration method. Equal-frequency binning (same number of samples per bin) is preferred over equal-width when predicted probabilities are clustered in narrow ranges.

CalibrationDisplay.from_predictions(y_true, y_prob) from sklearn.calibration plots the calibration curve. The x-axis is the mean predicted probability per bin; the y-axis is the fraction of actual positives in that bin. The dashed diagonal represents perfect calibration. Points above the diagonal mean the model is underconfident; points below mean it is overconfident.

from sklearn.calibration import CalibrationDisplay
import matplotlib.pyplot as plt

# y_true: actual labels, y_prob: predicted probabilities
CalibrationDisplay.from_predictions(y_true, y_prob, n_bins=10)
plt.plot([0, 1], [0, 1], 'k--', label='Perfect calibration')
plt.legend()
plt.show()

In multi-class settings, calibration is assessed per class using one-vs-rest reliability diagrams and per-class ECE, then macro-averaged across all K classes. Temperature scaling generalizes cleanly to multi-class: divide all logits by a single scalar T before softmax. The challenge is that multi-class calibration requires the full probability vector to be accurate, not just each class margin separately.

Platt scaling fits a sigmoid with only 2 parameters, making it stable and reliable when you have a small calibration set (hundreds of samples). Isotonic regression is non-parametric and more flexible but requires thousands of samples to avoid overfitting the calibration mapping. When uncertain, prefer Platt scaling for small datasets and isotonic regression when you have abundant held-out data.

Yes — calibration and discrimination are independent properties. A model that always predicts the base rate (e.g., P = 0.3 for every example in a class with 30% positives) is perfectly calibrated: on average, 30% of its predictions are indeed positive. But it has zero discriminative ability (AUC ≈ 0.5) and makes useless predictions for any individual decision. You need both calibration and discrimination for a model to be truly useful.

The Brier Score decomposes as BS = Uncertainty − Resolution + Reliability. Uncertainty is the irreducible variance of the data (fixed by the dataset, not the model). Resolution measures how much predictions deviate from the base rate — higher resolution means the model adapts its predictions to different cases (good). Reliability is the calibration error term — lower reliability penalty means better calibration.

Post-hoc calibration adjusts a trained model's output probabilities after training without modifying the model weights. Methods like Platt scaling, isotonic regression, or temperature scaling use a separate held-out calibration set. This is practical because it decouples expensive model training from probability calibration — you can calibrate any black-box model quickly and cheaply, and re-calibrate as data distributions shift.

A model that has been fitted to training data will produce overfit scores on those same examples — they are typically more extreme and confident than the true underlying probabilities. Using training data for calibration would produce a miscalibrated mapping that fails to generalize to new data. A held-out calibration set (or cross-validated predictions) must be used to get an honest, unbiased calibration.

Conformal prediction wraps any trained model to produce prediction sets (not just point probabilities) that are statistically guaranteed to contain the true label with coverage probability at least 1 − α, regardless of model type or data distribution. It uses a held-out calibration set to compute nonconformity scores and set a threshold. While it does not improve point probability accuracy, it provides a rigorous, distribution-free form of uncertainty quantification.

Logistic regression directly minimizes log-loss (cross-entropy), a proper scoring rule, so it naturally learns calibrated probability outputs. SVMs maximize the geometric margin and output raw decision scores (not probabilities) that need post-hoc calibration via Platt scaling. As a rule of thumb: logistic regression probabilities can be used directly; SVM scores should never be treated as probabilities without calibration.

When the calibration curve lies below the diagonal, the model's predicted probabilities are systematically higher than the actual positive rate — the model is overconfident. For example, if the model predicts P = 0.8 in a bin but only 55% of those examples are actually positive, that bin's point falls well below the diagonal. Overconfidence is common in decision tree ensembles, boosting methods, and uncalibrated neural networks.

For a K-class network with logits z, temperature-scaled softmax is p_i = exp(z_i/T) / Σ_j exp(z_j/T). When T = 1, this is standard softmax. T > 1 makes the distribution flatter and more uniform (less confident); T < 1 makes it more peaked (overconfident). The optimal T is found by minimizing validation log-loss after the network is fully trained, keeping accuracy unchanged.

import numpy as np

def temperature_softmax(logits, T):
    z = np.array(logits) / T
    e_z = np.exp(z - z.max())  # numerically stable
    return e_z / e_z.sum()

logits = [3.0, 1.0, 0.5]
print('T=1 (standard):', temperature_softmax(logits, 1).round(3))
print('T=2 (softer):  ', temperature_softmax(logits, 2).round(3))

In medicine, a model predicting P = 0.9 for a disease directly influences treatment decisions — a miscalibrated model could systematically over-treat or under-treat patients at scale. In finance, probability outputs drive portfolio sizing and risk management, where systematic biases lead to real losses. Unlike accuracy metrics that average over many samples, calibration ensures the probability number is meaningful for any single high-stakes individual decision.

Parametric methods (like Platt scaling) assume a fixed functional form (a sigmoid) with a small number of parameters (2), making them stable and reliable even with limited calibration data. Non-parametric methods (like isotonic regression) learn a free-form monotone mapping with no assumed shape — more flexible but prone to overfitting when calibration data is scarce. The right choice depends on the size of the held-out calibration set and the shape of the score distribution.