Testing and validating ML systems

Data leakage occurs when information that would not be available at prediction time (or that originates from the target) is used during training or feature construction, so the model effectively 'sees' the answer. It inflates offline metrics because the leaked signal does not generalize to production, producing optimistic validation scores that collapse in deployment. Common sources are fitting scalers/encoders on the full dataset before splitting, target-derived features, and temporal leakage where future data leaks into the training window.

Fitting transforms on the entire dataset leaks information from validation/test samples (their mean, variance, category vocabulary) into training, biasing estimates optimistically. The correct pattern fits the transform on the training fold and applies the learned parameters to the validation fold, which is exactly what a Pipeline guarantees during cross-validation.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
pipe = Pipeline([('sc', StandardScaler()), ('clf', LogisticRegression())])
scores = cross_val_score(pipe, X, y, cv=5)  # scaler refit per fold

Use forward-chaining whenever observations have a temporal ordering and the production task is to predict the future from the past. Standard k-fold shuffles data and lets future points train a model evaluated on past points, leaking temporal information and overstating performance. TimeSeriesSplit always trains on earlier data and validates on a strictly later window, mimicking real deployment.

Nested cross-validation uses an inner loop for hyperparameter tuning/model selection and an outer loop for unbiased performance estimation. It corrects the optimistic bias that arises when the same data is used both to select hyperparameters and to report the score, because tuning on the test folds leaks selection information. The outer-loop scores estimate generalization of the entire tuning procedure, not a single tuned model.

Stratified k-fold preserves the class proportions in each fold, ensuring every fold has a representative number of minority-class samples. Plain k-fold can produce folds with few or zero minority instances, yielding unstable, high-variance metric estimates and unreliable evaluation of minority-class performance.

GroupKFold ensures that all samples sharing a group identifier (e.g., the same patient, user, or device) appear in only one fold. It is required when correlated records would otherwise be split across train and test, letting the model memorize group-specific signal and leak it across the split. Without it, metrics overestimate performance on genuinely unseen groups.

An invariance test applies a label-preserving perturbation (e.g., a typo, paraphrase, or swapping a neutral named entity) and asserts the prediction should not change. A directional expectation test applies a perturbation expected to move the output in a known direction (e.g., adding strongly negative words should lower a sentiment score). Both are behavioral tests that probe model capabilities beyond aggregate accuracy.

Metamorphic testing checks relationships (metamorphic relations) between outputs of related inputs rather than checking each output against a known correct label. For example, scaling all features of a scale-invariant model should leave predictions unchanged, or adding a duplicate document to a search index should not lower the original's rank. It addresses the oracle problem: you can validate behavior without an explicit ground-truth oracle.

Train-serve skew is a discrepancy between the data/feature logic used at training time and what the model actually receives in production, caused by different code paths, feature versions, or distribution drift. Detect it by logging serving features and comparing their distributions to the training set, and by computing predictions on the same input through both pipelines to confirm parity. Shared feature transformation code (a feature store) is the structural fix.

Data drift (covariate shift) is a change in the input distribution P(X) while the relationship P(y|X) stays the same. Concept drift is a change in the underlying relationship P(y|X), so the same inputs now map to different outputs (e.g., changed user behavior or fraud tactics). Concept drift degrades accuracy even without input-distribution change and typically requires retraining on fresh labels, whereas data drift may be handled by reweighting or recalibration.

With severe imbalance a trivial majority-class predictor achieves high accuracy while being useless for the minority class, so accuracy hides poor minority performance. Better choices include precision-recall AUC, F-beta, balanced accuracy, Matthews correlation coefficient, and threshold-specific recall/precision aligned to the business cost. PR curves are especially informative because, unlike ROC, they are sensitive to the positive-class base rate.

A classifier is well-calibrated when its predicted probabilities match observed frequencies: among samples predicted at 0.7, about 70% are positive. Test it with a reliability diagram (binned predicted vs. observed frequency) and summary metrics like Expected Calibration Error (ECE) or Brier score. Miscalibration is common in modern deep nets and can be corrected post hoc with temperature scaling, Platt scaling, or isotonic regression.

Slice-based evaluation reports metrics separately on meaningful data subsets (e.g., by region, device, language, or demographic) rather than only the aggregate. Aggregate accuracy can mask severe underperformance on small but important slices, creating fairness and reliability failures. Tools like model cards and the SliceFinder approach surface these gaps so you can set per-slice acceptance gates.

A guardrail metric is a secondary metric you do not intend to improve but must not harm, such as latency, revenue, churn, or content diversity. It guards against a treatment that lifts the primary metric while damaging the broader product or user experience. You define acceptable degradation thresholds and stop or reject a variant that breaches them even if the primary metric wins.

Power analysis determines the sample size (or runtime) needed to detect a minimum effect of practical interest with a chosen significance level (alpha) and power (1-beta, often 0.8). Running underpowered tests wastes traffic and yields inconclusive or false-negative results, while it also prevents over-collecting. Inputs are baseline rate, minimum detectable effect, variance, alpha, and desired power.

Each decision made by looking at the validation score (model choice, threshold, hyperparameter) is an implicit fit to that set, so after many comparisons you adapt to its noise and the score becomes optimistic (the multiple-comparisons / adaptive-overfitting problem). Mitigate it with nested CV, a held-out test set touched only once, fresh test sets over time, or differential-privacy-style holdout reuse techniques like the reusable/Thresholdout holdout.

Use bootstrap resampling: repeatedly sample the test set with replacement, recompute the metric on each resample, and take percentiles (e.g., 2.5th and 97.5th) of the resulting distribution as the interval. This is non-parametric and works for metrics like AUC or F1 where closed-form variance is awkward. It quantifies evaluation uncertainty so you can tell whether two models differ meaningfully.

import numpy as np
from sklearn.metrics import roc_auc_score
rng = np.random.default_rng(0)
stats = []
for _ in range(2000):
    idx = rng.integers(0, len(y_true), len(y_true))
    stats.append(roc_auc_score(y_true[idx], y_score[idx]))
lo, hi = np.percentile(stats, [2.5, 97.5])

A golden set is a curated, version-controlled collection of high-value or historically failing examples with expected outputs, run in CI to catch regressions when code, data, or models change. Unlike the statistical held-out test set used to estimate generalization, the golden set is small, hand-picked, and acts as behavioral unit tests with hard pass/fail gates rather than aggregate metrics. The two are complementary, not interchangeable.

Contamination occurs when benchmark examples appear in the model's pretraining corpus, so high scores reflect memorization rather than generalization. Checks include n-gram/substring overlap searches between benchmarks and training data, canary strings deliberately embedded in datasets, perturbed or recently created held-out variants, and comparing performance on pre- vs post-cutoff data. Treat suspiciously high benchmark gains with skepticism until contamination is ruled out.

LLM judges exhibit position bias (favoring the first answer), verbosity/length bias, self-preference (favoring same-family outputs), and inconsistency across runs. Mitigations include randomizing/swapping answer order and averaging, using pairwise comparison with ties, calibrating against human labels, providing explicit rubrics with chain-of-thought, ensembling multiple judges, and reporting agreement metrics with humans. Always validate the judge on a human-labeled sample before trusting it at scale.

Offline metrics are computed on logged data collected under the old policy, so they suffer feedback loops and presentation bias and cannot observe how users react to items the old system never showed. Online A/B tests measure real user behavior under the new policy, capturing engagement effects, novelty, and interaction dynamics that offline replay misses. Counterfactual/off-policy estimators (e.g., inverse propensity scoring) help bridge the gap but rarely replace live tests.

In shadow (dark launch) deployment the new model receives a copy of live traffic and its predictions are logged but not served to users, so you compare behavior at production scale with zero user risk. In canary deployment the new model serves a small percentage of real traffic, exposing actual user impact while limiting blast radius before a full rollout. Shadow validates correctness/parity; canary validates real-world outcomes.

Schema/expectation checks codify assumptions about each feature: type, allowed range, expected categories, nullability, cardinality, and distribution. Run on incoming data, they catch missing columns, new/unseen categorical values, out-of-range or NaN spikes, and distributional skew before bad data reaches training or serving. They turn silent data corruption into loud, actionable test failures in the pipeline.

Like the software test pyramid, ML testing has many fast, cheap unit tests at the base (feature transforms, data schema, single-function logic), fewer integration tests in the middle (full pipeline, training reproducibility, serving parity), and few slow, expensive end-to-end/behavioral and online tests at the top. The principle is to push as much verification as possible to fast lower layers and reserve costly model-quality and live experiments for what only they can catch.

Beyond shape/NaN checks, test invariants and learned behavior: a single gradient step on a tiny batch should reduce loss; the model should overfit a handful of examples (sanity that it can learn); known directional relationships hold (raising a price lowers predicted demand); and outputs stay within valid bounds. These assert correctness of the modeling code and basic capability rather than aggregate accuracy.

def test_overfits_tiny_batch(model, tiny_X, tiny_y):
    opt = torch.optim.Adam(model.parameters(), lr=1e-2)
    for _ in range(200):
        opt.zero_grad(); loss = loss_fn(model(tiny_X), tiny_y)
        loss.backward(); opt.step()
    assert loss.item() < 1e-2  # model can fit a handful of points

When you run many simultaneous hypothesis tests (e.g., per-metric, per-slice, or many variants), the chance of at least one false positive grows with the number of tests. Bonferroni controls the family-wise error rate by dividing alpha by the number of tests (conservative), while Benjamini-Hochberg controls the false discovery rate (the expected proportion of false positives among rejections) and is less conservative for many tests. Without correction, peeking at many metrics manufactures spurious 'significant' wins.

Repeatedly checking a t-test and stopping as soon as p<0.05 inflates the false-positive rate far above the nominal alpha because each look is another chance to cross the threshold by noise. Fixes include fixing the sample size in advance and only testing once, or using sequential testing methods designed for continuous monitoring such as always-valid p-values, group-sequential boundaries (O'Brien-Fleming), or Bayesian/mSPRT approaches.

Backtesting evaluates a strategy/model on historical data by simulating decisions as they would have been made sequentially through time. Look-ahead bias occurs when information not yet available at the decision timestamp leaks into features or labels (e.g., using same-day close to predict same-day direction, or restated/revised data), producing unrealistically good backtest results. Proper backtests enforce point-in-time data and walk-forward (expanding/rolling) windows.

Adversarial testing crafts small, often imperceptible input perturbations (e.g., bounded L-infinity changes for images, character swaps or jailbreak prompts for text) to find inputs where the model fails despite high standard accuracy. It reveals brittleness and security vulnerabilities that average-case metrics hide, measuring worst-case rather than expected-case performance. Reporting robust accuracy under a defined threat model complements clean accuracy.

Mutation testing deliberately injects small faults (mutants) into the code or pipeline (e.g., flip a label, drop a feature, change an operator) and checks whether the existing tests catch them. It assesses the quality of your test suite itself: surviving mutants reveal blind spots where bugs would pass undetected. For ML it helps validate that data checks and model tests are actually sensitive to realistic defects.

Validating label quality means measuring annotation reliability (e.g., inter-annotator agreement via Cohen's/Fleiss' kappa), auditing samples, and detecting mislabeled instances (e.g., confident learning / Cleanlab). Label noise both degrades training and caps measurable performance: a noisy test set imposes a ceiling and can make a better model look worse, so reported accuracy reflects label errors, not just model errors. Cleaning or re-labeling the evaluation set is often the highest-leverage fix.

Reproducibility requires pinning random seeds, library/framework versions, data snapshots/hashes, and recording the full config and code commit. Full determinism on GPUs is hard because some parallel kernels (e.g., atomic reductions, certain cuDNN algorithms) are non-deterministic for speed, so identical inputs can yield slightly different outputs across runs. You enforce deterministic algorithm flags and accept a small performance cost when bit-exact reproducibility is required.

import torch, numpy as np, random
seed=42; random.seed(seed); np.random.seed(seed)
torch.manual_seed(seed); torch.cuda.manual_seed_all(seed)
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.benchmark = False

RAG evaluation must decompose the pipeline: retrieval quality (context precision/recall, hit rate, MRR/nDCG), grounding/faithfulness (is the answer supported by retrieved context, i.e., low hallucination), answer relevance to the query, and context utilization. Frameworks like RAGAS or TruLens score these separately so you can localize failures to the retriever versus the generator. Measuring only end-answer accuracy hides whether errors come from bad retrieval or bad generation.

In a champion-challenger setup the current production model (champion) serves traffic while one or more challenger models are evaluated either in shadow mode or on a small traffic slice. Performance is compared on live metrics over time, and a challenger is promoted only when it statistically beats the champion on primary metrics without breaching guardrails. This provides ongoing, low-risk validation and a clear, auditable promotion/rollback path as data drifts.