Regularization (L1, L2, dropout)

Regularization is a set of techniques used to reduce overfitting by discouraging a model from becoming too complex. It adds a penalty or constraint that pushes the model toward simpler patterns that generalize better to unseen data. The goal is to improve test/validation performance even if it slightly increases training error.

L2 regularization adds the sum of the squares of the model weights, multiplied by a coefficient lambda, to the loss function. This penalizes large weights, encouraging the model to keep weights small and spread out. It is also known as weight decay or ridge regularization.

loss = mse_loss + lam * sum(w**2 for w in weights)

L1 regularization adds the sum of the absolute values of the weights, multiplied by a coefficient lambda, to the loss. This penalty tends to push many weights exactly to zero, producing sparse models. It is also called lasso regularization.

loss = mse_loss + lam * sum(abs(w) for w in weights)

L1 regularization penalizes the absolute value of weights, and its constant-magnitude gradient keeps pushing weights toward zero even when they are already small. Because of the geometry of the absolute-value penalty (a diamond-shaped constraint with sharp corners), the optimal solution often lands exactly on an axis, setting some weights to exactly zero. This effectively performs automatic feature selection.

During training, dropout randomly 'drops' (sets to zero) a fraction of neurons in a layer on each forward pass, according to a chosen probability. This prevents neurons from co-adapting too strongly and forces the network to learn more robust, redundant representations. Different neurons are dropped on each mini-batch, so the network effectively trains many sub-networks.

No, dropout is turned off during inference; all neurons stay active. To keep the expected activation consistent with training, modern libraries use inverted dropout, which scales the surviving activations during training so that no extra scaling is needed at test time. This way the test-time network uses its full learned capacity.

Lambda is a hyperparameter that controls how strongly the regularization penalty influences training. A larger lambda increases the penalty, simplifying the model more (risking underfitting), while a smaller lambda lets the model fit the data more freely (risking overfitting). It is typically tuned using a validation set.

A very large lambda forces almost all weights toward zero, so the model becomes too simple and cannot capture the underlying patterns in the data. This leads to underfitting, where both training and test errors are high. The right lambda balances bias and variance.

Regularization increases a model's bias slightly while reducing its variance. By constraining the model, it becomes less sensitive to noise in the training data (lower variance) at the cost of being slightly less flexible (higher bias). The aim is to find the sweet spot that minimizes total error on unseen data.

In gradient descent, the L2 penalty adds a term proportional to each weight to its gradient. As a result, each update first multiplies the weight by a factor slightly less than one before applying the data gradient, so weights 'decay' toward zero over time. This shrinkage of weights gives the technique its name.

w = w - lr * (grad + 2 * lam * w)

L1 tends to drive many weights to exactly zero, producing sparse solutions useful for feature selection. L2 shrinks all weights smoothly toward zero but rarely makes them exactly zero, keeping all features with smaller magnitudes. So L1 selects features while L2 distributes weight more evenly.

Elastic Net combines both L1 and L2 penalties in a single loss function, controlled by separate (or blended) coefficients. It can produce sparse models like L1 while retaining L2's stability when features are correlated. This makes it useful when you want feature selection but also smooth shrinkage.

loss = mse + a*sum(abs(w) for w in weights) + b*sum(w**2 for w in weights)

This large gap indicates overfitting: the model has memorized the training set rather than learning general patterns. Regularization (such as L2, L1, or dropout) can help by constraining model complexity, which usually lowers training accuracy slightly but raises test accuracy. The goal is to shrink the train-test gap.

The bias term shifts the model's output independent of the input features, so penalizing it can prevent the model from fitting the correct baseline level of the target. Regularizing it offers little benefit for controlling complexity and can introduce unnecessary error. By convention, only the connection weights are regularized.

Each forward pass with dropout uses a different random subset of neurons, which is like training a different 'thinned' sub-network. Over many mini-batches, the procedure trains an exponential number of these overlapping sub-networks that share weights. At test time, using the full network with appropriately scaled activations approximates averaging the predictions of this ensemble.

Ridge regression applies L2 regularization, adding a squared-weight penalty controlled by the parameter alpha. Lasso applies L1, and ElasticNet combines both. Increasing alpha increases the strength of the L2 penalty.

from sklearn.linear_model import Ridge
model = Ridge(alpha=1.0)

Lasso regression uses L1 regularization, adding an absolute-value penalty that drives some coefficients to exactly zero. This makes Lasso useful for feature selection. The strength is controlled by the alpha parameter.

from sklearn.linear_model import Lasso
model = Lasso(alpha=0.1)

A dropout rate of 0.5 means that, on each training forward pass, each neuron in that layer has a 50% probability of being temporarily set to zero. This is a strong amount of dropout often used in large fully connected layers. Lower rates (e.g., 0.2) are common for less aggressive regularization.

Regularization penalizes weight magnitudes, so features on larger numerical scales would otherwise get unfairly large penalties relative to their importance. Standardizing features to similar scales ensures the penalty treats all features comparably. Without scaling, the regularization could distort which features are shrunk.

More training data gives the model a clearer picture of the true underlying pattern, making it harder to memorize noise from any single example. This reduces variance and overfitting, similar to what explicit regularization achieves. With enough data, a complex model can generalize well without strong penalties.

Early stopping monitors validation performance during training and halts when it stops improving, before the model starts overfitting the training data. By limiting how much the model adapts to training noise, it effectively constrains complexity. This makes it a simple and widely used implicit regularization method.

The L1 constraint forms a diamond (rotated square) with sharp corners lying on the coordinate axes. When the loss contours expand and first touch this region, they are likely to meet at a corner, where one or more coordinates are exactly zero. This geometry is why L1 yields sparse solutions, unlike the smooth circular L2 region.

The L2 penalty is lambda times the sum over all weights of w squared, often written as lambda * sum(w_i^2). Some formulations include a factor of 1/2 to simplify the gradient. This term is added to the base loss before optimization.

Yes, dropout and L2 (weight decay) are complementary and are frequently combined in deep networks. Dropout reduces co-adaptation among neurons, while L2 keeps individual weights small. Using both can give stronger regularization, though the hyperparameters may need retuning to avoid underfitting.

Co-adaptation happens when neurons become overly dependent on specific other neurons to correct their mistakes, creating fragile combinations that work only on training data. Dropout randomly removes neurons, so a neuron cannot rely on any particular partner being present. This forces each neuron to learn features that are useful on their own, improving robustness.

You call model.eval() before evaluating, which puts dropout (and batch norm) layers into inference mode so dropout is disabled. You switch back with model.train() for training. Forgetting model.eval() means dropout stays active during testing, producing noisy, inconsistent predictions.

model.eval()
with torch.no_grad():
    preds = model(x_test)

Weight decay (L2 regularization) is passed as the weight_decay argument when creating the optimizer, for example optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4). The optimizer then adds the corresponding shrinkage to each weight update. A value of zero disables it.

opt = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)

When two features are highly correlated, L1 tends to arbitrarily pick one and zero out the other, which can be unstable. L2 instead distributes the weight smoothly across the correlated features, giving each a share. This makes L2 more stable when features carry overlapping information.

L2 regularization is equivalent to placing a Gaussian (normal) prior with mean zero on the weights and finding the maximum a posteriori (MAP) estimate. The squared penalty comes from the log of a Gaussian density. In contrast, L1 corresponds to a Laplace prior, which favors sparsity.

A very high dropout rate removes most neurons on each pass, leaving too little capacity to learn the patterns in the data. This causes underfitting, where both training and validation accuracy stay low, and training becomes slow and unstable. Dropout rates are usually kept between about 0.2 and 0.5.

You try several candidate values (often on a logarithmic scale) and evaluate each using a validation set or cross-validation, then pick the value that gives the best validation performance. The training set alone cannot be used because stronger regularization always increases training error. Tools like GridSearchCV automate this search.

from sklearn.model_selection import GridSearchCV
GridSearchCV(Ridge(), {'alpha':[0.01,0.1,1,10]})

Inverted dropout scales the surviving activations by 1/(1-p) during training, where p is the drop probability, so the expected sum of activations stays the same. This means no scaling is needed at test time, keeping inference simple and fast. It is the standard implementation in modern deep learning libraries.

mask = (np.random.rand(*h.shape) > p) / (1 - p)
h = h * mask  # training only

L2 regularization shifts the optimal weights toward smaller magnitudes than the unregularized solution. The penalty term competes with the data-fit term, so the optimizer accepts slightly worse fit in exchange for smaller weights. This shrinkage is exactly what reduces variance and overfitting.

Data augmentation creates new training examples by applying label-preserving transformations (like rotating, flipping, or cropping images), exposing the model to more variation. This discourages the model from memorizing exact training pixels and encourages invariant, general features. The effect is reduced overfitting, similar to explicit regularizers.