Hyperparameter tuning

A hyperparameter is a configuration value set BEFORE training begins (e.g., learning rate, number of trees, k in k-NN), and it controls how the learning process works. In contrast, model parameters (like weights in linear regression) are learned automatically FROM the data during training. You tune hyperparameters manually or with search algorithms, but the model learns parameters on its own.

Hyperparameters control the learning process itself (such as model complexity), so optimizing them directly on training data would push toward configurations that overfit, like infinite tree depth or zero regularization. Instead, they must be evaluated on data the model did not train on (a validation set), so we can estimate generalization. This is why we use a separate validation set or cross-validation to tune them.

Grid search tries every possible combination from a predefined set of hyperparameter values, evaluating each one (usually with cross-validation) to find the best. Its main disadvantage is the curse of dimensionality: the number of combinations grows exponentially with the number of hyperparameters, making it very expensive when there are many to tune.

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
grid = {'C': [0.1, 1, 10], 'kernel': ['linear', 'rbf']}
search = GridSearchCV(SVC(), grid, cv=5)
search.fit(X_train, y_train)
print(search.best_params_)

Random search samples random combinations of hyperparameter values from specified distributions instead of testing every fixed combination. It is often more efficient because, when only a few hyperparameters truly matter, random sampling explores more distinct values of those important ones within the same budget. Bergstra and Bengio showed random search can find good configurations faster than grid search in high dimensions.

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import uniform
params = {'C': uniform(0.1, 10)}
search = RandomizedSearchCV(model, params, n_iter=20, cv=5)
search.fit(X_train, y_train)

The validation set is data held out from training that is used to evaluate different hyperparameter configurations and choose the best one. It acts as a proxy for unseen data, so we can estimate how well each configuration will generalize without touching the final test set. The test set is reserved for a single, unbiased evaluation only after tuning is complete.

k-fold cross-validation splits the data into k equal parts (folds); the model trains on k-1 folds and validates on the remaining one, repeating so each fold serves as validation once, then averaging the scores. It gives a more reliable estimate of performance for a given hyperparameter setting than a single split, reducing the variance caused by one lucky or unlucky split. This makes hyperparameter comparisons more trustworthy.

from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X, y, cv=5)
print(scores.mean())

If the learning rate is too high, the updates overshoot the minimum of the loss function, causing the loss to oscillate, diverge, or even grow toward infinity instead of converging. The model may never settle into a good solution. A learning rate that is too low, by contrast, converges very slowly and may get stuck.

Underfitting happens when a model is too simple to capture patterns (high bias), while overfitting happens when it is too complex and memorizes noise (high variance). Hyperparameters that increase complexity (e.g., deeper trees, more layers, less regularization) move a model from underfitting toward overfitting. Good tuning finds the sweet spot that minimizes total error on unseen data.

If you use the test set to choose hyperparameters, you indirectly fit your choices to that data, so the test score becomes overly optimistic and no longer reflects true generalization. This is a form of data leakage. The test set must remain untouched until the very end to give an honest, unbiased estimate of real-world performance.

Bayesian optimization builds a probabilistic surrogate model (often a Gaussian process) of the objective function that maps hyperparameters to validation performance, then uses an acquisition function to decide which configuration to try next. It balances exploring uncertain regions and exploiting promising ones, so it usually needs far fewer evaluations than grid or random search. This makes it valuable when each training run is expensive.

The regularization strength controls how strongly the model is penalized for having large weights, which limits model complexity. A larger value increases the penalty, shrinking weights toward zero and reducing overfitting (but risking underfitting), while a smaller value lets the model fit the data more freely. Tuning it balances bias and variance.

from sklearn.linear_model import Ridge
model = Ridge(alpha=1.0)  # alpha = regularization strength
model.fit(X_train, y_train)

A very small k (like k=1) makes the model highly sensitive to noise and individual points, leading to overfitting with a jagged decision boundary. A very large k smooths the decision boundary and can cause underfitting by ignoring local structure, potentially being dominated by the majority class. The best k is usually found by tuning on validation data.

Nested cross-validation uses an inner loop to tune hyperparameters and an outer loop to evaluate the resulting model's performance. It prevents the optimistic bias that occurs when the same data is used for both tuning and final evaluation, giving an unbiased estimate of how a tuning procedure generalizes. It is more computationally expensive but more honest.

Early stopping monitors validation performance during training and halts when it stops improving, preventing the model from continuing to fit noise. It effectively tunes the number of training epochs/iterations as a hyperparameter, choosing the point of best generalization. It is widely used in neural networks and gradient boosting to avoid overfitting.

These hyperparameters can span several orders of magnitude (e.g., 0.0001 to 1), and their effect on the model often changes multiplicatively rather than additively. Searching on a log scale (e.g., 1e-4, 1e-3, 1e-2, 1e-1) spreads the candidate values evenly across magnitudes, exploring the meaningful range efficiently. A linear scale would waste most samples in a narrow region.

import numpy as np
# Sample learning rates on a log scale
lrs = np.logspace(-4, -1, 4)  # [0.0001, 0.001, 0.01, 0.1]

max_depth limits how many levels of splits the tree can grow. A large or unlimited depth lets the tree create many fine splits that can memorize the training data and overfit, while a small depth keeps the tree simple and may underfit. Tuning max_depth controls the bias-variance trade-off.

from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier(max_depth=4)
model.fit(X_train, y_train)

There is a trade-off between computational cost and the quality of the solution found: evaluating more configurations increases the chance of finding a near-optimal setting but consumes more time and compute. With a limited budget you must balance exploring broadly versus refining promising regions. Smart methods like Bayesian optimization aim to get good results with fewer evaluations.

As you add more hyperparameters to a grid search, the total number of combinations grows exponentially, because it is the product of the number of values for each hyperparameter. For example, 5 hyperparameters with 10 values each gives 10^5 = 100,000 runs. This makes exhaustive grid search impractical for many hyperparameters.

In GridSearchCV, the number of model fits is fixed by the full grid (the product of all parameter value counts times the number of CV folds). In RandomizedSearchCV, you explicitly control the budget with n_iter, which sets how many random parameter combinations are sampled regardless of how large the search space is. This lets you cap the cost independently of search-space size.

RandomizedSearchCV(model, params, n_iter=50, cv=5)
# 50 random combos x 5 folds = 250 fits

Setting a fixed random seed makes processes like data splitting, weight initialization, and random sampling reproducible, so that performance differences you observe come from the hyperparameters rather than random chance. This allows fair, repeatable comparisons between configurations. Without it, the same settings could give different scores each run, confusing the comparison.

import numpy as np
np.random.seed(42)

Batch size and learning rate interact: larger batches give more stable, lower-variance gradient estimates, which often allows (and may require) a larger learning rate, while small batches produce noisier gradients that usually need a smaller learning rate for stability. A common heuristic is to scale the learning rate roughly with batch size. Both are tuned together because changing one affects the optimal value of the other.

Successive halving starts many hyperparameter configurations with a small resource budget (e.g., few epochs), then keeps only the top-performing fraction and gives them more resources, repeating until the best remains. Hyperband runs successive halving with different budget allocations to balance exploring many configs versus giving each enough resources. These methods efficiently discard poor configurations early, saving compute.

from sklearn.experimental import enable_halving_search_cv
from sklearn.model_selection import HalvingRandomSearchCV

best_estimator_ is the model that was retrained on the entire training set using the hyperparameter combination that achieved the highest mean cross-validation score. By default (refit=True), GridSearchCV automatically refits this best model so you can use it directly for predictions. It saves you from manually rebuilding the model with the best parameters.

search = GridSearchCV(model, grid, cv=5, refit=True)
search.fit(X_train, y_train)
best_model = search.best_estimator_

n_estimators sets the number of decision trees in the forest. Increasing it generally improves performance and stability up to a point because predictions are averaged over more trees, reducing variance, but beyond a certain number the accuracy gains plateau while training and prediction time keep increasing. So the trade-off is accuracy/stability versus computational cost.

from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=200)

Hyperparameter tuning can only optimize how well a model learns from the data it has, so once a reasonable configuration is found, further tuning yields small gains. In contrast, better features, more relevant data, or cleaning errors can fundamentally raise the ceiling of achievable performance. That is why practitioners often invest in data quality before exhaustive tuning.

C controls the trade-off between maximizing the margin and minimizing classification errors on the training data. A large C penalizes misclassifications heavily, leading to a narrower margin that fits the training data closely (risking overfitting), while a small C allows more margin violations for a wider, more general boundary (risking underfitting). It is an inverse regularization strength.

from sklearn.svm import SVC
model = SVC(C=1.0, kernel='rbf')

The search space is the set of hyperparameters and the ranges or distributions of values that a tuning algorithm is allowed to explore. Defining it well matters because too narrow a space may exclude the best settings, while too wide a space wastes the budget on unhelpful regions. Using domain knowledge to set sensible ranges (often on a log scale) makes the search much more effective.

The cv parameter sets the number of cross-validation folds, so each hyperparameter combination is trained and evaluated cv times. The total number of model fits is therefore (number of combinations) x cv, plus one final refit on all data. Increasing cv gives more reliable estimates but multiplies the computational cost.

# 6 combos x cv=5 = 30 fits (+1 refit)
GridSearchCV(model, grid, cv=5)

If you try a huge number of configurations and always pick the one that scores best on the same validation set, you can end up choosing settings that happen to suit that specific validation data by chance, not ones that truly generalize. This is overfitting to the validation set, and it makes validation scores overly optimistic. Using cross-validation, a separate final test set, or nested CV helps detect and reduce this.

In gradient boosting, a smaller learning rate makes each tree contribute less, which usually requires more estimators (boosting rounds) to reach the same fit, but often generalizes better. A large learning rate needs fewer trees but can overfit or be unstable. These two are tuned together, commonly by fixing a small learning rate and using early stopping to choose the number of estimators.

Manual tuning relies on a practitioner using intuition and experience to adjust hyperparameters one at a time and observe results, which can be insightful but slow and subjective. Automated tuning (grid search, random search, Bayesian optimization, etc.) systematically and reproducibly explores the search space using algorithms. Automated methods scale better and reduce human bias, though manual insight still helps define good search spaces.

Models like k-NN, SVM, and regularized regression are sensitive to feature scale, so unscaled features can distort distances and penalty terms, making hyperparameters like C, gamma, or k behave unpredictably. Scaling features (e.g., standardization) ensures hyperparameters operate on comparable footing, so tuning results are meaningful and transferable. Scaling should be done inside cross-validation (via a pipeline) to avoid leakage.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
pipe = Pipeline([('scaler', StandardScaler()), ('svc', SVC())])
GridSearchCV(pipe, {'svc__C':[0.1,1,10]}, cv=5)

Gamma defines how far the influence of a single training example reaches: a small gamma means each point influences a large region (smoother, simpler boundary), while a large gamma means influence is very local (more complex, wiggly boundary). A very large gamma can cause overfitting because the boundary tightly wraps individual points. Gamma is typically tuned together with C.

SVC(kernel='rbf', C=1.0, gamma=0.1)

Preprocessing must be fit only on the training folds because fitting it on the whole dataset (including validation folds) leaks information about the validation data into the model, producing overly optimistic scores. Using a Pipeline ensures the scaler or imputer is re-fit on each training fold and merely applied to the validation fold. This keeps the cross-validation estimate honest during hyperparameter tuning.

from sklearn.pipeline import make_pipeline
pipe = make_pipeline(StandardScaler(), LogisticRegression())
cross_val_score(pipe, X, y, cv=5)