Cross-validation and data splits

If we evaluate on the same data we train on, the model can simply memorize the examples and look highly accurate without truly learning generalizable patterns. A separate test set, kept unseen during training, estimates how well the model generalizes to new, real-world data. This gives an honest measure of performance.

The validation set is used during development to tune hyperparameters and compare models, guiding decisions without touching the test set. The test set is reserved for a single final evaluation after all choices are made. Keeping them separate prevents the test set from indirectly influencing the model and inflating the reported score.

The data is divided into k roughly equal folds; in each of the k rounds, one fold serves as the validation set while the remaining k-1 folds are used for training. Every fold is used for validation exactly once, and the final score is the average across all k rounds. This makes more efficient use of limited data than a single split.

from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X, y, cv=5)
print(scores.mean())

A single split gives one estimate that can be lucky or unlucky depending on which points land in the test set, especially with small data. K-fold averages performance over k different splits, producing a more stable and reliable estimate of generalization. It also uses every data point for both training and validation.

Stratified k-fold keeps the proportion of each class roughly the same in every fold as in the full dataset. This matters for imbalanced classification, where a random split might place too few (or zero) minority-class examples in some folds. Preserving class balance gives fairer, more representative scores.

from sklearn.model_selection import StratifiedKFold
skf = StratifiedKFold(n_splits=5)

LOOCV is k-fold cross-validation where k equals the number of data points, so each fold contains exactly one example. The model trains on all points but one and is tested on that single held-out point, repeated for every point. It uses almost all data for training but is computationally expensive and the per-fold scores can be highly variable.

Data leakage occurs when information from outside the training set (such as the test set or future data) influences the model, making evaluation overly optimistic. A common cause is fitting preprocessing steps like scaling on the whole dataset before splitting. The fix is to split first, then fit any transformations using only training data.

If you compute the mean and standard deviation using all data, the test set's statistics leak into preprocessing, giving an unrealistically good estimate. The correct approach fits the scaler on training data and then applies those same parameters to transform the test data. This mimics deployment, where future data is unknown when the scaler is built.

scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

Datasets are often ordered (for example, sorted by class or date), so taking the first chunk as training and the last as test could make the sets unrepresentative of each other. Shuffling mixes the examples so both sets reflect the overall distribution. Note: shuffling is wrong for time-series data, where order carries information.

Time-series observations are ordered in time, and random splitting would let the model train on future data to predict the past, which is unrealistic and causes leakage. Instead, you should use forward-chaining (time-based) splits where training always precedes the validation period. This respects the temporal order and gives a realistic estimate.

from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)

test_size sets the fraction (or absolute count) of the data placed in the test set; for example, test_size=0.2 reserves 20% for testing and uses 80% for training. It controls the trade-off between having enough training data and a reliable test estimate. A common choice is 0.2 to 0.3.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

random_state fixes the random number generator's seed so the shuffle and split are reproducible. Running the code again with the same seed yields the exact same split, which is essential for comparing experiments fairly and debugging. Without it, results would vary slightly each run.

The training set fits the model's parameters, the validation set tunes hyperparameters and selects models, and the test set provides one final, unbiased performance estimate. Keeping all three separate ensures the final score is not influenced by any tuning decisions. This is standard when enough data is available.

Nested cross-validation uses an inner loop to tune hyperparameters and an outer loop to estimate generalization performance. It prevents the optimistic bias that occurs when the same data is used both to choose hyperparameters and to report the score. It is more expensive but gives a more honest performance estimate.

A larger k means each training set is bigger, lowering the bias of the estimate, but it increases computation and the training sets overlap heavily, which can raise the variance of the estimate. A smaller k is faster but each model trains on less data, increasing bias. Common choices like k=5 or k=10 balance these concerns.

This large gap is a classic sign of overfitting: the model has memorized training-specific details and noise rather than learning generalizable patterns. It performs well on seen data but poorly on unseen data. Remedies include simplifying the model, adding regularization, or gathering more data.

If you repeatedly check the test set and adjust your model based on it, the test set effectively becomes part of tuning and its score no longer reflects true generalization. Using it just once keeps it a clean, unbiased estimate of real-world performance. All experimentation should happen on training and validation data.

Passing stratify=y makes the split preserve the class proportions of y in both the training and test sets. This is important for imbalanced datasets so that rare classes appear in both sets in representative amounts. Without it, a random split might leave a class underrepresented or absent in one set.

train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)

With little data, reserving a large fixed test set leaves too few examples for training and gives a noisy estimate. Cross-validation reuses every example for both training and validation across folds, so the full dataset contributes to both. This yields a more stable performance estimate without permanently sacrificing data.

GridSearchCV runs cross-validation on the data you pass to its fit method, typically the training data, trying each hyperparameter combination and averaging fold scores. The held-out test set should be kept aside and used only after the best parameters are chosen. This separation avoids leaking test information into tuning.

from sklearn.model_selection import GridSearchCV
gs = GridSearchCV(model, params, cv=5)
gs.fit(X_train, y_train)

The holdout method splits the data once into a training set and a separate held-out test set used only for evaluation. It is simple and fast but its estimate depends heavily on which examples land in each set, so it can be unreliable for small datasets. Cross-validation reduces this dependence.

Repeated k-fold runs the entire k-fold procedure multiple times with different random fold partitions and averages all the results. This reduces the variance caused by any single particular way of splitting into folds, giving a more robust estimate. The cost is extra computation proportional to the number of repeats.

from sklearn.model_selection import RepeatedKFold
rkf = RepeatedKFold(n_splits=5, n_repeats=3)

With 5 folds, each fold holds 1000/5 = 200 samples for validation, leaving the other 4 folds (800 samples) for training. So each of the 5 rounds trains on 800 samples and validates on 200. Across all rounds, every sample is validated exactly once.

The model has already seen the training examples and adjusted its parameters to fit them, so it naturally performs well on them, sometimes by memorizing rather than generalizing. This training accuracy does not reflect performance on new data. An independent test set is needed for an honest estimate.

A Pipeline ensures that preprocessing steps (like scaling) are fit only on each fold's training portion and then applied to that fold's validation portion. Because cross_val_score refits the whole pipeline per fold, the validation data never influences the preprocessing. This correctly mimics how unseen data would be handled.

from sklearn.pipeline import make_pipeline
pipe = make_pipeline(StandardScaler(), model)
cross_val_score(pipe, X, y, cv=5)

Duplicates across the split cause leakage: the model has effectively already seen the test examples during training, so the test score overestimates true performance. This is common with data that has repeated entries or grouped samples. Removing duplicates or using group-aware splitting prevents it.

GroupKFold ensures that all samples sharing the same group label (for example, all readings from one patient) stay together in the same fold and never appear in both training and validation. This prevents leakage when samples within a group are correlated. It gives a realistic estimate for data that arrives in natural groups.

from sklearn.model_selection import GroupKFold
gkf = GroupKFold(n_splits=5)

It is usually reported as the mean of the per-fold scores, often alongside the standard deviation to show how much performance varies across folds. The mean estimates expected performance, while the spread indicates stability and reliability. A high mean with a small spread is most desirable.

print(f"{scores.mean():.3f} +/- {scores.std():.3f}")

When the dataset is small, a single holdout split wastes scarce data and gives an estimate that swings widely depending on the split. Cross-validation reuses all data and averages over multiple splits, producing a far more reliable estimate. For very large datasets, a single holdout is often sufficient and cheaper.

A large spread suggests the model's performance is unstable and sensitive to which examples it trains on, often due to a small dataset, high model variance, or non-representative folds. It signals that a single estimate could be misleading and the model may not generalize consistently. More data or a simpler model can help.

If the test set is drawn from a different distribution than training (for example, different conditions or time period), the test score reflects distribution shift rather than the model's true learning. To fairly estimate generalization to the target population, both sets should represent the same underlying distribution. Otherwise conclusions about performance can be wrong.

By repeatedly optimizing against the test set, the student is effectively fitting to it, so the final test score no longer measures generalization and is optimistically biased. Hyperparameter tuning should be done with a validation set or cross-validation on training data. The test set must stay untouched until the final evaluation.

LOOCV trains the model n times, once for each sample held out as the single validation point. This uses the data efficiently but is computationally costly for large n. Each training run uses n-1 samples.

It returns an array of scores, one per fold, computed by training and evaluating the model on each split. You typically take the mean (and standard deviation) of this array to summarize performance. The scoring metric can be set with the scoring argument; otherwise the estimator's default is used.

scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')