Cheat sheet · Machine Learning

Hyperparameter tuning cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

Cross-validation (CV) gives a more reliable, lower-variance estimate of out-of-sample performance by reusing data across multiple train/validation splits (k-fold carries only a small pessimistic bias because each model trains on K-1/K of the data). Hyperparameter tuning searches the configuration space using CV (or a held-out set) as the objective. The core challenge is preventing any leakage of validation/test information into model selection so the reported score honestly predicts production performance.

Why CV & the Bias-Variance of Estimation

A single train/val split wastes data and gives a high-variance performance estimate; CV averages over K folds to reduce that variance while using all data for both training and evaluation.
k-fold splits data into K parts: train on K-1, validate on 1, rotate K times, average the metric. Typical K=5 or 10 (good bias-variance trade-off; K=10 is a common default).
Larger K -> lower bias (train sets closer to full size) but generally higher variance and more compute; LOOCV (K=n) is nearly unbiased but high-variance and expensive.
Repeated k-fold (different random seeds) further stabilizes the estimate and lets you report a mean +/- std or confidence interval.
CV estimates the performance of the *procedure* (model + tuning), not a single fixed model; the final production model is refit on all data with the chosen hyperparameters.
Report variance across folds, not just the mean: a high std signals instability, too-small folds, or a sensitive model.

Choosing the Right CV Scheme

StratifiedKFold for classification: preserves class proportions per fold, essential for imbalanced targets; use StratifiedGroupKFold when groups also matter.
GroupKFold when rows share an entity (patient, user, device): all rows of a group stay in one fold to prevent identity leakage and over-optimistic scores.
TimeSeriesSplit / rolling / expanding windows for temporal data: always train on the past, validate on the future, and NEVER shuffle time series (no peeking ahead).
Add an embargo/purge gap (purged k-fold, Lopez de Prado) when features/labels span overlapping windows to remove look-ahead leakage in finance/sequential data.
Nested CV when you both tune and estimate generalization: the outer loop estimates performance, the inner loop selects hyperparameters, avoiding the optimistic bias of tuning and evaluating on the same folds.
ShuffleSplit / Monte-Carlo CV for large data when you want a fixed number of random splits independent of K (folds may overlap; rows can repeat across validation sets).
For very large datasets, a single large held-out validation set can be enough and far cheaper than k-fold.

Preventing Leakage in the CV Loop

Fit ALL preprocessing (scaling, imputation, encoding, feature selection, SMOTE/resampling, target encoding) INSIDE each training fold; fitting on the full dataset before CV leaks test stats.
Use an sklearn Pipeline so transforms are refit per fold automatically; pass the Pipeline (not pre-transformed X) to cross_val_score / GridSearchCV.
Resampling (SMOTE, undersampling) must happen only on the training fold; use imblearn.pipeline.Pipeline and never oversample before splitting.
Feature selection driven by the target must be inside CV; selecting top-k features on the whole set then CV-ing is a classic leaked-score bug.
Keep a final, untouched test set (or a nested-CV outer loop) that is touched exactly once; repeated peeking turns it into a validation set and inflates results.
Group/time leakage: duplicate rows, near-duplicates, or future-derived features silently inflate scores even with otherwise correct splitting.

Hyperparameter Search Strategies

Grid search: exhaustive over a discrete grid; simple and reproducible, but cost grows exponentially with the number of params (curse of dimensionality).
Random search: sample configurations from distributions; beats grid for the same budget because few hyperparameters usually matter (Bergstra & Bengio, 2012). Prefer log-uniform for learning rates / regularization.
Bayesian optimization (Optuna, Hyperopt TPE, scikit-optimize): builds a surrogate of the objective and uses an acquisition function (EI/UCB) to focus on promising regions, sample-efficient for expensive models.
Successive Halving / Hyperband (ASHA): allocate small budgets to many configs, promote the top performers; great for deep nets and large data (multi-fidelity). sklearn ships HalvingGridSearchCV / HalvingRandomSearchCV (enable via enable_halving_search_cv).
Optuna adds pruning (stop unpromising trials early), define-by-run search spaces, and parallel distributed trials; a 2026 default for serious tuning alongside Ray Tune.
Use early stopping on a validation fold for boosting / NNs to tune the number of iterations 'for free' rather than as a grid axis.
Always tune against the CV score (or a proper held-out set), and set/track random seeds for reproducibility.

Nested CV, Selection & Final Model

Nested CV: outer K folds give an unbiased generalization estimate; an inner CV/search within each outer training split picks hyperparameters. The gold standard when data is limited.
Flat (non-nested) CV that reports the best inner score as the generalization estimate is optimistically biased: the score was maximized over many configs.
After selecting hyperparameters, refit a single model on ALL available training data with those settings for deployment.
Don't blindly pick the absolute best CV mean: prefer the simplest model within one standard error of the best (the 'one-standard-error rule') to reduce overfitting to validation noise.
Multiple-comparison risk: testing hundreds of configs against the same CV folds can win by chance; more trials need more validation rigor or a final hold-out confirmation.
Calibrate probabilities (Platt/isotonic) on held-out CV folds, not on data the model already trained on (use CalibratedClassifierCV).

Metrics, Budget & Practical Tuning

Pick a scoring metric aligned with the business goal: ROC-AUC vs PR-AUC (imbalance), F-beta, log-loss/Brier (probability quality), RMSE/MAE/MAPE (regression). Tuning to the wrong metric optimizes the wrong thing.
Use the same metric for CV, tuning, and reporting; mismatches produce models that look good but fail on the real objective.
Coarse-to-fine: a wide random/Bayesian search first, then refine around the best region; use log scales for learning rate and regularization strength.
Budget the search: random/Hyperband/Bayesian beat grid under fixed compute; parallelize trials and reuse cached folds where possible.
Watch for overfitting the validation set: a large gap between CV score and final test score means too many trials or leakage.
For deep learning, tune learning rate first (highest impact), then batch size/schedule/regularization; use a fixed val split, or k-fold only if data is small.

Common pitfalls

Fitting scalers/imputers/encoders/feature-selectors or SMOTE on the full dataset before CV, leaking test information and inflating scores.
Shuffling time-series data or using random k-fold on temporal/sequential problems (look-ahead leakage).
Letting rows from the same entity (user/patient/group) fall in both train and validation folds (identity leakage).
Reporting the best inner-CV/tuning score as the generalization estimate (optimistic bias) instead of using nested CV or a hold-out.
Reusing the test set repeatedly during development until it silently becomes a validation set.
Using accuracy (or the wrong metric) on imbalanced data, and not stratifying classification folds.
Confusing CV (which estimates a procedure) with producing a single model, and forgetting to refit on all data after tuning.
Over-searching: hundreds of trials on the same folds can win by chance (multiple comparisons) with no final confirmation.
Not fixing random seeds, making CV scores and tuning results irreproducible.
Choosing the single best config by tiny CV-mean differences while ignoring fold variance / the one-standard-error rule.

Interview tips

Explain the bias-variance trade-off of K (small vs large K, LOOCV) and why K=5/10 is standard.
Expect 'how do you prevent data leakage in CV?' Answer with Pipelines refit per fold and concrete examples (scaling, SMOTE, target encoding, feature selection).
Know when to use Stratified vs Group vs TimeSeries vs purged/embargoed CV and justify each by the data structure.
Explain nested CV: why flat CV is optimistically biased and what the outer vs inner loops each estimate.
Compare grid vs random vs Bayesian (TPE) vs Hyperband/successive-halving and argue which fits a given compute budget.
Explain why random search often beats grid (only a few hyperparameters matter; effective dimensionality is low).
Choose and defend a scoring metric for imbalanced classification (PR-AUC/F-beta over accuracy) and discuss probability calibration.
Address reproducibility, the one-standard-error rule, and refitting the final model on all data before deployment.
Mention modern tooling (Optuna pruning/define-by-run, ASHA, Ray Tune, sklearn Halving search) and multi-fidelity / early-stopping tricks.
Handle the trap: 'you tuned 500 configs, CV looks great but test is worse, why?' (overfitting the validation set / leakage / multiple comparisons).

Read the full lesson Test yourself — 34 question quiz