Overfitting and regularization

Overfitting occurs when a model learns the training data too closely, including its noise and random fluctuations, rather than the underlying pattern. As a result, it performs very well on training data but poorly on new, unseen data. The model has high variance and fails to generalize.

Underfitting happens when a model is too simple to capture the underlying structure of the data, so it performs poorly on both training and test data. Unlike overfitting (high variance), underfitting is characterized by high bias. Overfitting fits the training data too well while underfitting fits it too poorly.

The bias-variance tradeoff describes the balance between two sources of error: bias (error from overly simple assumptions, causing underfitting) and variance (error from sensitivity to training data, causing overfitting). Reducing one often increases the other, so the goal is to find a model complexity that minimizes total error. Good generalization comes from balancing both.

Regularization is a set of techniques that add a penalty for model complexity to discourage overly complex models. It is used to prevent overfitting by keeping model parameters small or sparse, which improves generalization to unseen data. Common examples include L1 and L2 regularization.

L2 regularization adds the sum of squared weights (multiplied by a factor lambda) to the loss function. This penalizes large weights and shrinks them toward zero, but rarely makes them exactly zero. It produces smoother, more stable models that are less sensitive to individual data points.

L1 regularization adds the sum of the absolute values of weights to the loss, which can drive some weights exactly to zero. This produces sparse models and performs automatic feature selection. L2, by contrast, shrinks weights smoothly but keeps most of them nonzero.

Lambda controls the strength of the regularization penalty. A larger lambda places more emphasis on keeping weights small, increasing bias but reducing variance; a smaller lambda lets the model fit the data more freely, risking overfitting. When lambda is zero, regularization has no effect.

As overfitting begins, training error continues to decrease while validation error starts to increase. The growing gap between the two curves is a classic sign of overfitting. The ideal stopping point is where validation error is at its minimum.

Early stopping monitors performance on a validation set during training and halts training when the validation error stops improving (and begins to rise). This prevents the model from continuing to fit noise in the training data. It is a simple and effective form of regularization for iterative learners like neural networks.

Dropout is a regularization technique where, during training, a random fraction of neurons is temporarily 'dropped' (set to zero) in each forward pass. This prevents neurons from co-adapting too much and forces the network to learn more robust, redundant features. At test time, all neurons are used (with appropriate scaling).

More training data gives the model a broader and more representative view of the true underlying pattern, making it harder to memorize noise specific to a small sample. With more examples, random fluctuations average out and the model is pushed toward learning genuine structure. This generally lowers variance and improves generalization.

A validation set is data held out from training and used to evaluate the model during development. By comparing performance on training versus validation data, we can detect overfitting (high training accuracy but low validation accuracy). It also guides hyperparameter tuning without touching the final test set.

K-fold cross-validation splits the data into k equal parts, trains on k-1 folds and validates on the remaining fold, repeating this k times so each fold serves as validation once. The results are averaged to give a more reliable estimate of performance. This reduces the chance that a lucky or unlucky single split misleads us about generalization.

Increasing model complexity gives the model more flexibility to fit the training data, which lowers bias but raises variance. Beyond a certain point, the extra flexibility starts fitting noise, causing overfitting and worse generalization. The goal is to choose a complexity that balances fit and simplicity.

Perfect training accuracy can indicate that the model has memorized the training data, including its noise, rather than learning generalizable patterns. Such a model may perform poorly on unseen data. The real test is performance on a separate validation or test set.

Occam's razor states that simpler explanations are generally preferable. Regularization embodies this principle by penalizing complexity, encouraging the model to prefer simpler hypotheses that still explain the data well. Simpler models tend to generalize better and are less prone to overfitting.

You import Ridge from sklearn.linear_model and instantiate it with the alpha parameter, which controls the L2 penalty strength. A larger alpha means stronger regularization. You then fit it like any other estimator with the .fit() method.

from sklearn.linear_model import Ridge
model = Ridge(alpha=1.0)
model.fit(X_train, y_train)

Data augmentation artificially expands the training set by applying label-preserving transformations to existing data, such as rotating, flipping, or cropping images. This exposes the model to more varied examples, encouraging it to learn invariant features rather than memorizing specifics. It effectively increases data diversity and reduces overfitting.

Generalization means the model performs well on new, previously unseen data, not just the data it was trained on. A model that generalizes well has captured the true underlying patterns rather than noise. Good generalization is the ultimate goal of supervised learning.

The test set provides an unbiased estimate of how the model will perform on truly unseen data. If it were used during training or repeated tuning, the model would indirectly adapt to it, leaking information and inflating the estimate. Keeping it untouched until the end preserves the integrity of the evaluation.

A high-variance model produces very different predictions and parameters when trained on slightly different datasets, because it is highly sensitive to the specific training samples. This instability is a hallmark of overfitting. Reducing variance through regularization or more data makes the model more consistent.

Elastic Net combines both L1 and L2 penalties in a single model, controlled by a mixing parameter. It benefits from L1's feature selection (sparsity) and L2's stability and handling of correlated features. This makes it useful when there are many features, some of which are correlated.

Irrelevant features give the model extra degrees of freedom to fit noise and spurious correlations that happen to appear in the training data. This increases variance and can hurt generalization, since the model may rely on patterns that don't hold in new data. Feature selection or L1 regularization can help mitigate this.

The L1 penalty creates a diamond-shaped (corner) constraint region, and the corners lie on the axes where some coefficients are exactly zero. The optimal solution often touches these corners, setting those weights to zero. L2's circular constraint region has no corners, so it rarely produces exact zeros.

For a fixed model, increasing the dataset size primarily reduces variance, since the model has more examples to estimate patterns reliably. Bias generally stays about the same because it depends on the model's inherent assumptions. Thus more data mainly helps overfitting (high-variance) problems, not underfitting.

Pruning removes branches of a decision tree that add little predictive power, simplifying the tree. A fully grown tree can perfectly fit (and overfit) the training data, so pruning reduces complexity and improves generalization. It is a form of regularization specific to tree models.

Limiting maximum depth restricts how many splits the tree can make, preventing it from creating overly specific rules that fit individual training points. A shallower tree captures broad patterns rather than noise, improving generalization. It directly controls model complexity.

Parameters are learned from data during training, such as the weights of a regression model. Hyperparameters are set before training and control the learning process, such as the regularization strength lambda (or alpha). We tune hyperparameters using a validation set, not the training loss alone.

If there is a large gap between low training error and high validation error that persists as data grows, the model is overfitting (high variance). If both errors are high and close together, the model is underfitting (high bias). The shapes of these curves guide whether to add data, reduce complexity, or increase complexity.

Regularization penalizes the magnitude of weights, so features on larger scales would otherwise get unfairly small penalties relative to their importance, distorting the result. Scaling features (e.g., standardization) ensures the penalty is applied fairly across all features. Without scaling, regularization may shrink some coefficients inappropriately.

If lambda is too high, the penalty dominates the loss and forces weights toward zero, making the model too simple. This causes underfitting, with high error on both training and validation data. The model loses the flexibility needed to capture real patterns.

Random Forest trains many decision trees on bootstrapped samples with random feature subsets, then averages (or votes on) their predictions. This ensembling reduces variance because the errors of individual overfit trees tend to cancel out. The result generalizes much better than any single deep tree.

The bias term shifts the model's output and represents the baseline prediction when all features are zero. It is typically left unregularized because penalizing it would unfairly force predictions toward zero rather than toward the data's true mean. Regularization is meant to control feature weights, not the overall offset.

In LogisticRegression, C is the inverse of regularization strength, so a small C means strong regularization and a large C means weak regularization. This is the opposite of alpha in Ridge/Lasso, where larger means stronger. Choosing C controls the bias-variance tradeoff.

from sklearn.linear_model import LogisticRegression
# Small C = strong regularization
model = LogisticRegression(C=0.1)
model.fit(X_train, y_train)

Grid search tries a range of candidate lambda (or alpha) values, evaluating each with cross-validation to estimate generalization performance. The value giving the best average validation score is selected. This systematically finds the regularization strength that best balances bias and variance.

from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import Ridge
params = {'alpha': [0.01, 0.1, 1, 10, 100]}
grid = GridSearchCV(Ridge(), params, cv=5)
grid.fit(X_train, y_train)
print(grid.best_params_)

If the deployment data differs from the training distribution (distribution shift), patterns learned during training may not apply, causing poor real-world performance even if validation looked good. The model may have over-relied on features specific to the training environment. This shows generalization depends on the test data resembling the training data.

Noise refers to random, non-informative variation in data, such as measurement errors or irreducible randomness in the labels. Overfitting happens when a model treats this noise as if it were a real pattern and fits it. Because noise differs in new data, fitting it harms generalization.

From a Bayesian perspective, L2 regularization corresponds to assuming a Gaussian (normal) prior centered at zero on the weights. This expresses a belief that small weights are more likely, and the penalty pulls estimates toward that prior. L1 regularization similarly corresponds to a Laplace prior.

This pattern indicates overfitting, where the model fits the training data including its noise but fails to generalize. Remedies include adding regularization, getting more training data, reducing model complexity, or using techniques like dropout or early stopping. The aim is to reduce variance.

Batch normalization normalizes layer inputs using statistics from each mini-batch, which introduces a small amount of noise because the batch statistics vary between batches. This noise acts somewhat like regularization, making the network less sensitive to specific inputs and slightly reducing overfitting. However, its main purpose is to stabilize and speed up training.