Model calibration quiz

Question 1

What does it mean for a machine learning model to be 'calibrated'?

Accepted Answer

Ensuring predicted probabilities match actual outcome frequencies

Answer

Training a model on more data to improve accuracy

Answer

Scaling feature values to the range [0, 1]

Answer

Adjusting the learning rate during training

Question 2

What is a reliability diagram (calibration curve), and what does the diagonal line represent?

Accepted Answer

A plot of predicted probabilities vs actual positive rates across bins

Answer

A plot of precision vs recall for binary classifiers

Answer

A histogram of model errors on training data

Answer

A ROC curve for evaluating ranking quality

Question 3

What does it mean when a model is overconfident, and how does it appear on a calibration curve?

Accepted Answer

The model predicts probabilities too extreme, closer to 0 or 1 than reality warrants

Answer

The model predicts probabilities too close to 0.5

Answer

The model has high variance and changes predictions frequently

Answer

The model always predicts the majority class label

Question 4

What is the Brier Score and how is it computed?

Accepted Answer

Mean squared error between predicted probabilities and actual binary outcomes

Answer

Mean absolute error between predicted class labels and true labels

Answer

Area under the ROC curve for binary classification

Answer

Sum of squared residuals from a linear regression

Question 5

Why is log-loss considered a calibration-sensitive metric?

Accepted Answer

Log-loss is minimized when predicted probabilities match true class probabilities

Answer

Log-loss only measures the rank order of predictions, not their magnitude

Answer

Log-loss is insensitive to overconfident wrong predictions

Answer

Log-loss equals accuracy when the classification threshold is 0.5

Question 6

How does Platt scaling calibrate a classifier's output?

Accepted Answer

Fits a logistic regression on raw classifier scores using a validation set to produce calibrated probabilities

Answer

Retrains the model with a regularization penalty directly on probability outputs

Answer

Replaces each prediction with the bin average from training data frequencies

Answer

Normalizes output scores by dividing each by the maximum score across the dataset

Question 7

What is isotonic regression calibration, and when should you prefer it over Platt scaling?

Accepted Answer

Isotonic regression fits a non-decreasing piecewise function and needs more calibration data than Platt scaling

Answer

Isotonic regression fits a polynomial curve to predicted scores

Answer

Isotonic regression is a parametric method with fewer parameters than Platt scaling

Answer

Isotonic regression retrains the model from scratch using calibration loss

Question 8

What is Expected Calibration Error (ECE) and how is it calculated?

Accepted Answer

The weighted average absolute gap between bin accuracy and bin confidence across M bins

Answer

The average log-loss across all predictions in the test set

Answer

The standard deviation of all predicted probability values

Answer

The mean squared error between AUC and accuracy on the test set

Question 9

Why is Naive Bayes often poorly calibrated even when it achieves good accuracy?

Accepted Answer

The feature independence assumption pushes probabilities to extremes near 0 and 1, causing overconfidence

Answer

Naive Bayes has too few parameters to model the complexity of real data

Answer

Naive Bayes uses softmax output which is inherently miscalibrated

Answer

Naive Bayes ignores the prior probability of each class during prediction

Question 10

Why does gradient boosting (e.g., XGBoost) tend to produce poorly calibrated probabilities?

Accepted Answer

Gradient boosting optimizes for ranking/residual loss rather than probability accuracy, producing overconfident outputs

Answer

Gradient boosting uses dropout which randomizes its probability outputs

Answer

Gradient boosting averages too many trees, making predictions underconfident and uniform

Answer

Gradient boosting does not output probabilities at all — only class labels

Question 11

How does CalibratedClassifierCV in scikit-learn calibrate any classifier?

Accepted Answer

It wraps a classifier and applies Platt or isotonic calibration on cross-validated held-out predictions

Answer

It averages predictions from multiple base classifiers trained on independent random subsets

Answer

It retrains the model after multiplying input features by calibration weights

Answer

It only works with logistic regression as the base model

Question 12

What is temperature scaling in neural network calibration, and how is T chosen?

Accepted Answer

T divides logits before softmax and is tuned on a validation set to soften or sharpen outputs

Answer

T is trained alongside model weights using backpropagation on the training set

Answer

T scales the input features before they are fed into the network layers

Answer

Temperature scaling is a data augmentation technique that varies training examples

Question 13

What is the difference between model calibration and model discrimination?

Accepted Answer

Discrimination measures ranking quality; calibration measures probability accuracy — a model can have high AUC but poor calibration

Answer

Discrimination and calibration measure the same property but on different datasets

Answer

Calibration measures training speed; discrimination measures test set accuracy

Answer

A well-calibrated model always achieves high AUC by definition

Question 14

What does underconfidence look like on a calibration curve?

Accepted Answer

Predictions cluster near 0.5; the calibration curve lies above the diagonal because actual positive rates exceed the model's conservative predicted probabilities

Answer

The calibration curve lies entirely and consistently below the diagonal

Answer

The model always predicts exactly 0.5 for every single input example

Answer

The model assigns negative probability values to some examples

Question 15

What is Maximum Calibration Error (MCE) and how does it differ from ECE?

Accepted Answer

MCE is the maximum bin-level calibration gap; ECE is the weighted average — MCE is stricter for safety-critical use

Answer

MCE is the average calibration error over all bins; ECE is the maximum error

Answer

MCE measures accuracy on edge cases; ECE measures calibration for the positive class only

Answer

MCE and ECE are identical metrics known by different names in different papers

Question 16

How does class imbalance affect a model's calibration?

Accepted Answer

Imbalance causes models to underestimate minority class probabilities, skewing the calibration curve

Answer

Class imbalance has no effect on calibration since it is computed on probabilities, not counts

Answer

Imbalance causes the model to over-predict probabilities for the majority class

Answer

Class imbalance only affects precision and recall, not probability calibration

Question 17

What is 'sharpness' in probabilistic prediction, and how does it relate to calibration?

Accepted Answer

Sharpness means predictions are extreme (near 0 or 1); desirable only when the model is also calibrated

Answer

Sharpness means the model always predicts the same probability for all examples

Answer

Sharpness is equivalent to precision in binary classification tasks

Answer

A sharp model is automatically well-calibrated by definition

Question 18

What is a proper scoring rule, and why does it encourage calibration? Give two examples.

Accepted Answer

A proper scoring rule is minimized only when predicted probabilities match true probabilities; log-loss and Brier Score are examples

Answer

A proper scoring rule penalizes only false positives, not false negatives

Answer

Accuracy is the most common proper scoring rule used in practice

Answer

Proper scoring rules only apply to binary classification, not multi-class problems

Question 19

How does label smoothing reduce overconfidence and improve model calibration?

Accepted Answer

Label smoothing replaces hard 0/1 targets with soft values, preventing extreme logits and reducing overconfidence

Answer

Label smoothing removes noisy training labels before model fitting begins

Answer

Label smoothing adds Gaussian noise to input features to improve generalization

Answer

Label smoothing decreases the learning rate near convergence to avoid overshooting

Question 20

Why are modern deep neural networks often poorly calibrated despite achieving high accuracy?

Accepted Answer

Deep networks with batch normalization and high capacity become overconfident — softmax outputs don't reflect true uncertainty despite high accuracy

Answer

Deep networks use ReLU activations which clip probabilities at zero, causing systematic miscalibration

Answer

Deep networks are miscalibrated because they do not use cross-entropy loss during training

Answer

Deep networks are well-calibrated whenever validation accuracy exceeds 90%

Question 21

What is histogram binning for probability calibration?

Accepted Answer

Histogram binning replaces each bin's predicted probability with the observed positive fraction from a validation set

Answer

Histogram binning removes predictions that fall outside the [0.1, 0.9] probability range

Answer

Histogram binning retrains the model on a stratified sample to balance class counts

Answer

Histogram binning converts continuous regression outputs into discrete class probabilities

Question 22

Which scikit-learn function plots a calibration curve, and how do you interpret it?

Accepted Answer

sklearn.calibration.CalibrationDisplay.from_predictions — the diagonal represents perfect calibration

Answer

sklearn.metrics.roc_curve — the diagonal there represents random guessing

Answer

matplotlib.pyplot.calibration_plot — a built-in matplotlib calibration utility

Answer

sklearn.metrics.plot_confusion_matrix — used to assess probability quality

Question 23

How is calibration evaluated in multi-class classification?

Accepted Answer

Multi-class calibration uses per-class one-vs-rest ECE and temperature scaling on logits, macro-averaged across classes

Answer

Multi-class calibration is evaluated by computing accuracy for each class separately with a 0.5 threshold

Answer

Multi-class calibration only applies to balanced datasets with equal class frequencies

Answer

Temperature scaling cannot be applied to multi-class problems — only to binary classifiers

Question 24

How should you choose between Platt scaling and isotonic regression for post-hoc calibration?

Accepted Answer

Use Platt scaling for small calibration sets (2 parameters, stable) and isotonic regression for large sets (flexible)

Answer

Always use isotonic regression since it makes no assumptions and is always more accurate

Answer

Choose whichever method gives higher accuracy on the training set

Answer

Platt scaling only works for SVMs; isotonic regression works for all model types

Question 25

Can a perfectly calibrated model have poor accuracy or poor discriminative ability?

Accepted Answer

Yes — a model always predicting the base rate is perfectly calibrated but has zero discriminative ability

Answer

A perfectly calibrated model always achieves at least 80% accuracy on balanced datasets

Answer

No — calibration guarantees accuracy since both measure prediction quality

Answer

Perfect calibration implies AUC = 1.0 by definition

Question 26

What is the Brier Score decomposition and what does each component mean?

Accepted Answer

BS = Uncertainty − Resolution + Reliability; resolution (higher is better) and reliability/calibration (lower is better) are the key components

Answer

BS = Bias + Variance + Noise, where Bias is fixed by the data distribution

Answer

BS = Precision × Recall − Calibration Error, summarizing three quality aspects

Answer

BS = Log-loss + Accuracy Error, two parts that can be minimized independently

Question 27

What is post-hoc calibration and why is it a practical approach in production ML?

Accepted Answer

Post-hoc calibration adjusts output probabilities after training using a held-out set — no model retraining needed

Answer

Post-hoc calibration modifies the model architecture before training begins

Answer

Post-hoc calibration fine-tunes all model weights on the validation set to minimize calibration loss

Answer

Post-hoc calibration is only valid for deep learning models, not classical ML methods

Question 28

Why should you never calibrate a model using the same data it was trained on?

Accepted Answer

Models overfit to training data so their scores are not representative of true probabilities — a held-out set is required

Answer

Training sets cannot be used because they are too small for calibration methods to work

Answer

Training sets only contain positive examples, biasing the calibration curve

Answer

Scikit-learn prevents calibration on the training set by design through an internal check

Question 29

What is conformal prediction and how does it relate to the concept of calibration?

Accepted Answer

Conformal prediction produces prediction sets with guaranteed coverage probability using a calibration set — a rigorous form of uncertainty quantification

Answer

Conformal prediction improves raw probability outputs using isotonic regression on residuals

Answer

Conformal prediction calibrates neural networks by learning a softmax temperature parameter

Answer

Conformal prediction removes low-confidence predictions from model output to improve reliability

Question 30

How does logistic regression compare to SVMs in terms of probability calibration, and why?

Accepted Answer

Logistic regression outputs calibrated probabilities naturally; SVMs output margin scores and need post-hoc calibration

Answer

SVMs are better calibrated than logistic regression because they use margin-based training

Answer

Both models produce equally calibrated outputs since both use sigmoid-based decision boundaries

Answer

Logistic regression is poorly calibrated because gradient descent overfits training probabilities

Test yourself