Dropout and preventing overfitting

Overfitting occurs when a model learns the training data too well — including its noise and random quirks — so it performs excellently on training examples but poorly on new, unseen data. Think of it like memorising exam answers word-for-word instead of understanding the concepts: you'd struggle with any question that was phrased differently. A telltale sign is very high training accuracy paired with much lower validation accuracy.

This is a classic case of underfitting: the model is too simple (high bias) to learn the underlying patterns, so both training and test accuracy are roughly the same — and both are poor (barely better than random for 10 classes). Underfitting means the model has not even mastered the training data, let alone generalised to new data. The fix is to increase model capacity, train longer, or engineer better features.

As model complexity increases, bias decreases because the model can express more complex patterns, but variance increases because the model becomes sensitive to the specific training examples it saw. A single-neuron linear model has high bias (can't fit curves) but low variance (stable predictions). A very deep network has low bias but high variance — it can overfit. The total error is Bias² + Variance + irreducible noise, and the goal is to minimise their sum.

Dropout is a regularisation technique where, during each training step, each neuron in a designated layer is independently switched off (output set to zero) with probability p (the dropout rate). This stops any single neuron from becoming so dominant that the network relies on it exclusively. Crucially, dropout is only active during training — at inference (prediction) time all neurons are switched back on so the full network is used.

During training with dropout rate p, on average only (1−p) of neurons are active, so activations are smaller than if all neurons were on. If we then run inference with all neurons active, outputs would be (1/(1−p)) times larger than expected. Inverted dropout solves this by multiplying surviving activations by 1/(1−p) during training, so the expected value is the same as having all neurons on — meaning at test time no adjustment is needed and the full network runs as-is.

Dropout(0.3) means each neuron in the preceding layer has a 30% chance of being set to zero on every training step (70% chance of surviving). Typical recommendations are 0.2–0.5 for fully-connected hidden layers, with 0.5 often cited as a strong default and 0.1–0.2 for convolutional layers where spatial structure already provides some redundancy. Setting it too high can cause underfitting because the effective network becomes too small to learn.

import tensorflow as tf
model = tf.keras.Sequential([
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.3),   # 30% dropout
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(10, activation='softmax')
])

Co-adaptation is when groups of neurons work together to memorise specific training patterns — if neuron A always fires together with neuron B, they can jointly encode noise. Because dropout can remove either neuron at any training step, each neuron must learn features useful on its own without relying on partners. This forces the network to build multiple redundant representations of each concept, resulting in richer and more generalisable features.

Surviving activations are divided by k = 0.8, which is the same as multiplying them by 1/0.8 = 1.25. Without this, the expected sum of activations would be 0.8 times smaller than at inference (where all neurons are on), causing a mismatch. By scaling up during training, the expected activation magnitude stays the same at both training and test time, so the network can be used at inference without any modification.

import numpy as np

def inverted_dropout(A, keep_prob=0.8):
    mask = (np.random.rand(*A.shape) < keep_prob)  # True = keep
    A = A * mask / keep_prob  # scale surviving neurons up
    return A

L2 regularisation adds the term λ × Σw² to the training loss, where λ controls how strongly large weights are penalised. Because the optimiser tries to minimise this combined loss, it is pushed to keep weights small, preventing any single connection from becoming so strong it encodes noise. In the weight update rule, L2 adds a term −2ηλw to each update, which gradually shrinks every weight toward zero each step — hence the name 'weight decay'.

L1 regularisation adds λ × Σ|w| (sum of absolute values) to the loss, while L2 adds λ × Σw² (sum of squares). The key difference is that L1's gradient is a constant ±λ regardless of the weight's size, so even tiny weights keep getting pushed and eventually hit zero — producing sparse models with many zero weights. L2's gradient is proportional to w, so large weights are shrunk fast but small weights are only nudged gently and rarely reach exactly zero. L1 therefore acts as automatic feature selection.

The L2 penalty adds a term 2λw = 2 × 0.01 × 2.0 = 0.04 to the gradient, giving a total gradient of 0.5 + 0.04 = 0.54. The weight update is w_new = w − η × total_gradient = 2.0 − 0.1 × 0.54 = 2.0 − 0.054 = 1.946. The weight has been nudged down both by the data gradient and the regularisation penalty, which discourages large weights.

Early stopping means halting training when the model's performance on a held-out validation set stops improving, even if the training loss is still decreasing. The divergence between a falling training loss and a rising (or plateauing) validation loss signals that the model has begun memorising training data rather than learning general patterns. The best practice is to save model weights at the epoch where validation loss is lowest and restore them after stopping, giving you the most generalisable model.

Data augmentation artificially expands the training set by applying label-preserving transformations to existing images, so the model sees more diverse versions of each class and is less likely to memorise any particular pixel pattern. Common transformations include random horizontal flipping, random cropping, and colour jitter (adjusting brightness or contrast). By training on augmented data the model learns invariances — that a cat flipped horizontally is still a cat — which directly reduces overfitting.

from tensorflow.keras.preprocessing.image import ImageDataGenerator

datagen = ImageDataGenerator(
    rotation_range=20,
    horizontal_flip=True,
    zoom_range=0.2,
    width_shift_range=0.1
)

Batch normalisation normalises each layer's activations to zero mean and unit variance across a mini-batch, then applies learnable scale (γ) and shift (β) parameters; its primary purpose is to stabilise and speed up training, though it adds mild regularisation as a side effect. Dropout directly targets overfitting by injecting random noise (zero-ing neurons) each step. They can be used together, though in practice batch norm often reduces the need for heavy dropout — CNNs commonly use batch norm with light dropout rather than heavy dropout alone.

In k-fold cross-validation the data is split into k equal subsets (folds); the model is trained k times, each time using k−1 folds for training and the remaining fold for validation, and the k validation scores are averaged. A single split can be lucky or unlucky depending on which examples ended up in which part; averaging over k different validation sets gives a much more stable estimate of how well the model really generalises. If training accuracy is consistently high while cross-validation accuracy is low, overfitting is confirmed regardless of which particular split you used.

The training set is used to optimise model weights via gradient descent. The validation set is used to tune hyperparameters (learning rate, dropout rate, number of layers) and to monitor for overfitting during development — because decisions are made based on validation performance, the model is indirectly influenced by it. The test set is kept completely hidden until the very end to provide an unbiased, final estimate of real-world performance; using it earlier would leak information and make the reported accuracy optimistic.

Overfitting shows up as a growing gap between the two curves: training loss keeps decreasing toward a very low value while validation loss first decreases, then levels off or starts rising. This means the model is memorising training examples rather than learning generalisable rules. A well-trained model would show both curves converging to similar low values; if both remain high, that indicates underfitting instead.

When you add λ×Σw² to the loss and use Adam, the regularisation gradient (2λw) gets divided by Adam's adaptive per-parameter scale (√v̂ + ε), which weakens the penalty for parameters with large gradient history. AdamW (decoupled weight decay) bypasses Adam's scaling by applying the decay w ← w(1 − ηλ) directly to the weights, independent of gradients — every weight shrinks by the same proportion each step. This makes regularisation far more consistent and predictable, which is why AdamW generally outperforms Adam + L2 loss on large models.

A single deep decision tree tends to overfit by memorising training data down to individual examples. Bagging trains many trees, each on a random bootstrap sample of the data (drawn with replacement), then combines predictions by averaging (regression) or majority vote (classification). Because each tree makes different errors on its sample and those errors are largely uncorrelated, averaging reduces the variance dramatically — even though each individual tree still overfits its own subset. Random Forests add further diversity by considering only a random subset of features at each split.

Dropout is most effective in large fully-connected (dense) hidden layers, where co-adaptation between neurons is the biggest risk. It is generally not applied to the input layer (dropping raw features removes real information), the output layer (adding noise to final predictions is harmful), or immediately after batch normalisation layers (which already normalise statistics and interact poorly with dropout stochasticity). In convolutional layers standard dropout is less effective; spatial dropout (dropping entire channels) is used there instead.

In a convolutional feature map, adjacent pixels are highly correlated because they were computed using overlapping receptive fields. If one pixel is dropped, the surrounding pixels can still pass the same information forward, making standard per-pixel dropout almost useless. Spatial dropout (2D dropout) solves this by dropping an entire feature map (channel) at once — if the feature map for 'horizontal edge detector' is dropped, no information from that detector reaches the next layer, creating strong regularisation pressure.

MC Dropout keeps dropout active at test time and runs N forward passes through the network (e.g., N = 100); each pass randomly drops different neurons, producing slightly different predictions. The mean of the N predictions gives the model's best estimate, while the variance across them quantifies uncertainty — high variance means the model 'disagrees with itself' and is uncertain. This technique turns a standard dropout network into a cheap approximation of a Bayesian neural network, letting it say 'I don't know' instead of always giving a confident single output.

# Keep training=True at inference for MC Dropout
predictions = [model(X_test, training=True) for _ in range(100)]
mean_pred = np.mean(predictions, axis=0)
uncertainty = np.var(predictions, axis=0)  # high = uncertain

Max-norm regularisation constrains the L2 norm of the incoming weight vector for each neuron to be at most a threshold c: if ||w|| > c, rescale w so that ||w|| = c. Unlike L2, which continuously nudges all weights downward at every step, max-norm does nothing when ||w|| ≤ c and only acts at the boundary. This makes it robust even with high learning rates because weights can never blow up past c, and it was often paired with dropout in early deep learning research for complementary coverage.

Standard dropout zeros out entire neuron activations, effectively removing all connections to that neuron simultaneously. DropConnect is more fine-grained: it randomly zeros individual weights (connections) in the weight matrix, keeping all neuron activations alive. Dropout is therefore a special case of DropConnect where all weights leading to a dropped neuron are zeroed at once. DropConnect creates an even larger implicit ensemble of networks (one per weight-mask pattern) but is more computationally expensive to implement.

Input noise injection adds small Gaussian noise to training inputs, teaching the model to be robust to measurement errors. Weight noise randomly perturbs weight values during each forward pass, which is theoretically equivalent to a form of regularisation on the loss landscape. Label smoothing replaces hard one-hot targets with soft distributions (e.g., 0.9 for the correct class, 0.033 for each other class), preventing the model from becoming overconfident and improving calibration. Together these techniques address overfitting from different angles.

Label smoothing replaces a hard one-hot vector like [0, 0, 1, 0] with a soft version that gives a small probability ε/(C−1) to each wrong class and (1−ε) to the correct one. With ε = 0.1 and 4 classes this becomes [0.033, 0.033, 0.9, 0.033]. Training with cross-entropy against these soft targets penalises the model for pushing the logit of the correct class to +∞, preventing overconfident outputs. This improves generalisation and calibration because the model learns to retain a small degree of uncertainty even on training examples it has seen many times.

During model.fit(), Keras passes training=True internally to every layer, so Dropout(0.5) randomly zeros 50% of the neurons in the 256-unit and 128-unit layers on every forward pass. During model.predict() (or model.evaluate()), Keras passes training=False, and Dropout layers automatically become identity functions — all neurons are active and the full network is used. Because Keras uses inverted dropout internally, no manual output scaling is needed at prediction time.

model = tf.keras.Sequential([
    Dense(256, activation='relu'),
    Dropout(0.5),
    Dense(128, activation='relu'),
    Dropout(0.5),
    Dense(10, activation='softmax')
])

With a 0.7 dropout rate only 30% of neurons survive each step, making the effective network very small — training accuracy will drop substantially. If the original model was genuinely overfitting, validation accuracy might improve at first as regularisation kicks in, but too high a rate can push the model into underfitting where both training and validation accuracy suffer. Training also becomes slower because the highly thinned network has less gradient signal per step and needs more epochs to converge.

The Lottery Ticket Hypothesis (Frankle & Carlin, 2019) states that within a large randomly-initialised network there exist small sub-networks ('winning tickets') that, when trained from their original initialisations, can match the full network's test accuracy. This means overparameterisation is useful not because every neuron is needed in the final model, but because a large random initialisation increases the probability of containing a good winning ticket. It also explains why training a randomly pruned small network from scratch usually fails — you need the specific initial values of the winning ticket.

With more training data the model is exposed to greater variety, making it harder to memorise any specific subset and naturally improving the signal-to-noise ratio. Techniques like dropout and L2 regularisation work by constraining the model's capacity to fit noise, but they are compensating for too little data rather than solving the root problem. More data lets you use a more powerful model without overfitting and tends to give larger accuracy gains than any regularisation tweak. In practice, however, labelled data is expensive to collect, so regularisation is essential when data is limited.

With 20 neurons and 50/50 keep probability, each neuron is independently included or excluded, giving 2²⁰ ≈ 1,048,576 possible binary masks and therefore over a million distinct subnetworks. Each training step samples and trains one of these subnetworks. At test time the full network with scaled weights approximates the geometric mean of predictions from all 2²⁰ subnetworks simultaneously — a massive ensemble that is vastly cheaper than training millions of models separately. This ensemble effect is a key reason dropout is such an effective regulariser.

L1 regularisation penalises the absolute values of weights (Σ|w|), encouraging sparsity — many weights become exactly zero, which is useful for feature selection. L2 regularisation penalises squared weights (Σw²), shrinking all weights proportionally but rarely to exactly zero, and is a solid general-purpose regulariser. Dropout, by contrast, does not penalise weights at all; it reduces overfitting by zeroing random neuron outputs during training, effectively training an ensemble. The three techniques can be combined and target overfitting through fundamentally different mechanisms.

The 37-percentage-point gap between training and validation accuracy is a clear sign of overfitting: the model has far more parameters than 500 examples can constrain. With only 500 training examples, a 10-layer network can simply memorise every sample instead of learning general patterns. First corrective steps are: reduce model depth or width, add dropout (e.g., 0.3–0.5) to fully-connected layers, apply L2 regularisation, and if the data is images add augmentation. Getting more labelled data would be the best long-term fix.

For each candidate λ, train a model with all other hyperparameters fixed and record the validation set loss or accuracy after training; then select the λ that gives the best validation performance. A very small λ (0.0001) may still overfit; a very large λ (0.1) may underfit by shrinking weights too aggressively. The test set must not be used here because every time you peek at test-set performance to make a decision, you are implicitly tuning your model to that set, causing optimistic bias — the reported test accuracy would no longer be a fair estimate of real-world performance.