Early stopping and checkpoints

Early stopping is a regularization technique that halts training when the model's performance on a validation set stops improving, preventing overfitting. Instead of training for a fixed number of epochs, training stops automatically when a monitored metric (like validation loss) no longer improves after a set number of epochs. This saves compute time and preserves the model at its best generalization point.

Validation loss (val_loss) is typically monitored during early stopping because it measures how well the model generalizes to unseen data. When validation loss starts increasing while training loss continues to decrease, it signals overfitting. Using validation loss rather than training loss prevents the model from continuing to train once generalization degrades.

Patience is the number of consecutive epochs with no improvement in the monitored metric before training is stopped. For example, patience=5 means training will continue for up to 5 more epochs after the last improvement, giving the model a chance to recover from temporary fluctuations. A higher patience value allows more time for potential improvement but risks longer unnecessary training.

A model checkpoint is a saved snapshot of a model's weights (and optionally its optimizer state) at a specific point during training. Checkpoints allow you to resume training if it is interrupted, compare models at different stages, or restore the best-performing model after training completes. They are essential for long training runs where saving progress periodically prevents losing hours of compute work.

Using the test set to make early stopping decisions would cause the model to indirectly 'see' test data, making the final evaluation biased and overly optimistic. The validation set is held out from weight updates and used only to guide decisions like early stopping and hyperparameter tuning. The test set must remain completely untouched until final evaluation to give a true estimate of real-world performance.

Early stopping limits the number of training steps, which restricts how much the model's weights can move away from their initialization values. This implicitly constrains the model's complexity, similar to how L2 regularization penalizes large weights. Research shows that for quadratic loss functions, early stopping in gradient descent is mathematically equivalent to L2 (weight decay) regularization under certain conditions.

A patience of 1 means training stops after just one epoch without improvement, which is too aggressive. Validation loss naturally fluctuates due to mini-batch randomness and stochastic gradient descent noise, so a single non-improving epoch is often a temporary dip rather than a true plateau. Setting patience too low risks stopping training prematurely before the model has reached its best generalization point.

The 'last' checkpoint saves weights at the final training epoch, which may be an overfitted state if training ran too long. The 'best' checkpoint saves the model at the epoch where the monitored metric (e.g., val_loss) was optimal. For deployment and evaluation, the 'best' checkpoint is almost always preferred because it represents peak generalization performance rather than the endpoint of training.

When training loss continues to decrease but validation loss starts to increase (diverging curves), it signals overfitting and that early stopping should have triggered around the validation loss minimum. The optimal stopping point is roughly where validation loss is at its lowest before it begins rising. This classic divergence pattern — training improving while validation degrades — is the primary signal early stopping is designed to detect.

When restore_best_weights=True, Keras automatically reverts the model's weights to those from the epoch where the monitored metric was best, after early stopping triggers. Without this option, the model would retain the weights from the last (potentially worse) epoch. This ensures you end up with the optimal model without needing a separate ModelCheckpoint callback to save and reload the best weights manually.

Option A is correct: it monitors 'val_loss' (the right metric for detecting overfitting), uses patience=10 to tolerate 10 non-improving epochs, and sets restore_best_weights=True to revert to the best checkpoint after stopping. Option B monitors training loss using the Keras key 'loss' — this is wrong because training loss almost always decreases monotonically, giving no useful overfitting signal. Option C has patience=0 (stops after the very first non-improving epoch), and Option D uses patience=1 which is overly aggressive.

from tensorflow.keras.callbacks import EarlyStopping

# Option A
es_A = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)

# Option B  (Keras key for training loss is 'loss', not 'train_loss')
es_B = EarlyStopping(monitor='loss', patience=10, restore_best_weights=False)

# Option C
es_C = EarlyStopping(monitor='val_accuracy', patience=0, restore_best_weights=True)

# Option D
es_D = EarlyStopping(monitor='val_loss', patience=1, restore_best_weights=True)

min_delta sets the minimum change in the monitored metric that qualifies as an improvement. For example, min_delta=0.001 means the metric must improve by at least 0.001 to reset the patience counter. Without min_delta, even a trivially small improvement (like 0.000001) would reset patience and extend training unnecessarily; min_delta filters out insignificant changes and makes early stopping more robust.

This code saves the model to 'best_model.keras' only when validation loss reaches a new minimum (save_best_only=True, mode='min'). Each time validation loss improves, the file is overwritten with the new best model weights. This ensures the saved file always contains the best-performing version seen during training, regardless of whether training continues or eventually overfits.

from tensorflow.keras.callbacks import ModelCheckpoint

cp = ModelCheckpoint(
    filepath='best_model.keras',
    monitor='val_loss',
    save_best_only=True,
    mode='min'
)

Early stopping terminates training entirely when improvement plateaus, while ReduceLROnPlateau reduces the learning rate to allow finer optimization without stopping. Learning rate scheduling tries to push past plateaus by taking smaller gradient steps, whereas early stopping accepts the current best model and halts. They are complementary: ReduceLROnPlateau can help escape plateaus first, and then EarlyStopping terminates training when even reduced steps cannot improve performance.

When training loss decreases but validation loss increases, the model is memorizing training-specific patterns (noise, outliers) that do not generalize to new data — this is overfitting. The model's weights become too tightly tuned to the training set, capturing its idiosyncrasies rather than the true underlying pattern. This divergence is the classic signal that early stopping should trigger to preserve the best generalizing model.

For a quadratic loss function (like mean squared error) optimized with gradient descent at a small learning rate, early stopping is approximately equivalent to L2 regularization. The number of training steps T plays the role of inverse regularization strength: fewer steps ≈ stronger regularization (λ ≈ 1/T). This relationship is formally discussed in Goodfellow, Bengio, and Courville's Deep Learning textbook (2016), Chapter 7 — fewer gradient steps keep weights closer to their initialization, mimicking L2 weight penalties.

Early stopping prevents the high-variance (overfitting) extreme by stopping before the model perfectly memorizes training data. A model trained too long achieves very low training error (low bias) but poor generalization (high variance). Early stopping finds a middle ground: accepting slightly higher training error (marginally more bias) in exchange for much better generalization to new data (significantly lower variance).

Early stopping can hurt performance when the validation loss curve is very noisy — with large epoch-to-epoch fluctuations — and the true minimum occurs deep in training but is temporarily masked by spikes. It can also be harmful when the validation set is too small, making the stopping signal unreliable. In such cases, using a larger patience, more validation data, loss smoothing, or k-fold cross-validation are better strategies.

mode='min' means a lower metric value is better — used for loss metrics like val_loss where you want the value to decrease. mode='max' means a higher metric value is better — used for accuracy or F1 where you want the value to increase. Setting the wrong mode would cause early stopping to never trigger correctly (e.g., mode='max' on val_loss would think higher loss is an improvement and never stop).

The early stopping criterion is evaluated at the end of each epoch, after the model has processed the entire training dataset and the validation loss has been computed on the held-out validation set. This epoch-level evaluation is standard because it provides a stable signal compared to within-epoch batch-level checks, which would be noisy and potentially misleading during learning.

Checkpointing too rarely risks losing significant training progress if a crash or power failure occurs mid-run, forcing costly retraining from scratch. Checkpointing too frequently creates large storage overhead since each checkpoint saves all model weights, which can be multiple gigabytes for large models. The typical balance is to checkpoint every N epochs or only when a new best metric is achieved, trading off storage cost against recovery safety.

Batch normalization computes running mean and variance statistics over training batches, and these statistics may still be shifting when early stopping triggers prematurely. If training halts before these running statistics have stabilized, the batch norm layers may perform poorly at inference time when they switch from batch statistics to the accumulated running statistics. Careful patience tuning can help ensure batch norm statistics are stable before training stops.

Training loss almost always decreases monotonically as the model trains more, because the model directly optimizes for training data. Using it as the early stopping criterion means early stopping would effectively never trigger — training would run for all max_epochs. The purpose of early stopping is to detect when generalization degrades, which only the held-out validation set can reveal; training loss is blind to this.

The ??? should be torch.save(model.state_dict(), 'best_model.pth') to save the model's weight dictionary when validation loss improves. This is the standard PyTorch pattern: save state_dict (a plain dictionary of tensors, not the full model object) on improvement, and later reload with model.load_state_dict(torch.load('best_model.pth')). The logic correctly resets the patience counter on improvement and increments it otherwise until patience is exhausted.

best_val_loss = float('inf')
patience_counter = 0
patience = 5

for epoch in range(max_epochs):
    train(model)
    val_loss = evaluate(model, val_loader)

    if val_loss < best_val_loss:
        best_val_loss = val_loss
        patience_counter = 0
        ???  # Save the best model
    else:
        patience_counter += 1

    if patience_counter >= patience:
        print('Early stopping triggered')
        break

ReduceLROnPlateau reduces the learning rate when the monitored metric stops improving, allowing the optimizer to take finer gradient steps and potentially escape a plateau. EarlyStopping then monitors for further improvement and terminates training if even the reduced learning rate cannot help. Together they form a two-stage strategy: first try smaller steps to improve (ReduceLROnPlateau), then stop if nothing works (EarlyStopping) — often producing better final models than either callback alone.

After EarlyStopping triggers, training halts and no further weight updates occur. If restore_best_weights=True, Keras reverts the model's weights to those from the epoch with the best monitored metric, undoing the recent epochs of degradation. If restore_best_weights=False (the default), the model retains the weights from the last training epoch, which may be slightly worse than the optimal checkpoint.

By plotting training and validation loss over epochs, you can observe how quickly validation loss typically recovers after temporary dips. If the validation curve shows fluctuations that resolve within 3-5 epochs, a patience of 5-10 is appropriate. If the model shows longer improvement cycles (recovering after 15-20 flat epochs), patience should be higher. Learning curves thus provide data-driven intuition for setting patience rather than guessing blindly.

Dropout randomly zeros activations during each forward pass, encouraging each neuron to learn independently and preventing co-adaptation — it acts within each training step. Early stopping limits the total number of training iterations, constraining how far all weights collectively deviate from initialization. They target overfitting through different mechanisms: dropout disrupts co-dependence per sample while early stopping controls the total complexity budget. Both are often used together for stronger regularization.

With patience=1000, early stopping effectively never triggers within a reasonable training run, removing its protective effect against overfitting. On a small dataset where overfitting happens quickly (often within tens of epochs), this allows the model to train far into the overfitting regime, yielding poor generalization. Patience should be calibrated to the dataset size and training dynamics — smaller datasets need shorter patience because overfitting occurs faster.

The 'best checkpoint' saves the model with optimal validation performance for final deployment, while the 'latest checkpoint' (most recent epoch) enables resuming training from exactly where it left off in case of crashes or infrastructure failures. Without the latest checkpoint, a crash would require retraining from scratch or from the best checkpoint, potentially redoing many stable training epochs. Together they balance fault-tolerance (latest) with quality preservation (best).

Epoch-level checkpointing saves the model once per epoch — after the entire training dataset has been processed — providing stable, consistent snapshots. Batch-level (step-level) checkpointing saves after every N gradient update steps within an epoch, which is valuable for very long epochs with millions of steps where losing a full epoch of progress would be costly. Most Keras callbacks default to epoch-level checkpointing, which is sufficient for most standard training runs.

Noisy validation curves can cause early stopping to trigger prematurely during a temporary spike, even if the overall trend is still improving. Effective mitigation strategies include: increasing patience to outlast short-term noise, using min_delta to require meaningful improvements, ensuring a larger validation set for more stable per-epoch estimates, and checking that training data is shuffled properly each epoch. The root causes of a noisy validation curve are typically a small validation set or insufficient data shuffling.

The baseline parameter sets a minimum metric value that must be reached before early stopping begins monitoring for stagnation. For example, baseline=0.90 with val_accuracy means early stopping won't activate until accuracy exceeds 90%, preventing premature stopping during the initial rapid-learning phase when any fluctuation might otherwise look like a plateau. Once the baseline is crossed, normal patience-based monitoring begins.

Storing only a reference (e.g., best_weights = model.state_dict()) can be unsafe because PyTorch optimizers typically update parameter tensors in-place during each step. If the saved reference shares underlying storage with the live parameters, subsequent gradient updates silently overwrite the 'saved' values. A deep copy (copy.deepcopy(model.state_dict())) creates a fully independent snapshot of weight values frozen at that moment, guaranteed to be immune to any subsequent in-place gradient updates.

import copy

# RISKY: may reflect later in-place optimizer updates
best_weights_bad = model.state_dict()

# CORRECT: independent snapshot frozen at this epoch
best_weights_good = copy.deepcopy(model.state_dict())

# Later, restore the best weights:
model.load_state_dict(best_weights_good)