Cross-Validation Explained: How K-Fold Works and Why It Beats a Single Split

Every model needs to be judged on data it hasn't seen — that part everyone knows. The subtler problem is that a single train/test split is a lottery: shuffle differently and your accuracy can swing by several points, because which examples happen to land in the test set matters. Cross-validation fixes this by making *every* example take a turn in the test set, then averaging the results into one steadier, more trustworthy estimate.

How k-fold cross-validation works

1. Shuffle the dataset and split it into k equal parts, called folds (k = 5 or 10 is standard).
2. Train the model on k−1 folds and evaluate it on the one held-out fold.
3. Repeat k times, so each fold serves as the test set exactly once.
4. Average the k scores — that average is your performance estimate, and their spread tells you how stable it is.

Choosing k — and the variants that matter

• k = 5 or 10 is the practical default: a good balance of reliability and compute cost.
• Stratified k-fold keeps the class ratio identical in every fold — essential for imbalanced classification, so no fold ends up with barely any positives.
• Leave-one-out (k = n) trains n models, each missing a single point — nearly unbiased but expensive and often high-variance; mainly for very small datasets.
• Time-series split — for temporal data you must always train on the past and test on the future. Ordinary shuffled k-fold lets the model peek ahead, which inflates scores.

The leakage mistakes that inflate your scores

Cross-validation is only honest if each fold's test data stays truly unseen. The classic mistake is preprocessing before splitting: if you fit a scaler, imputer or feature selector on the whole dataset, information from the test folds leaks into training, and your scores drift optimistically. The fix is to put every fitted step inside the cross-validation loop (a pipeline), so it's re-fit on each training fold only — the same fit-on-train-only rule we cover in feature scaling. The second classic mistake: tuning hyperparameters on the same folds you report results from. Use nested cross-validation, or keep a final untouched test set for the last word — the same discipline behind hyperparameter tuning.

Cross-validation is also your first line of defence for diagnosing overfitting vs underfitting: a large gap between training scores and CV scores is the overfitting signature. For deeper drills, see the cross-validation and data splits topic page. And **AI Learning** lets you practice all of it — evaluation, splits, leakage traps — through ~10,000 offline questions with explanations, no account needed.

AI Learning: AI/ML/DS Q&A

Free · Works offline · Master evaluation and validation with spaced practice questions.

Coming soon toGoogle Play