Learning rate schedules

A learning rate schedule is a rule or policy that changes the learning rate during training instead of keeping it fixed. It typically starts with a larger learning rate to make fast progress and reduces it over time so the model can settle into a good minimum. Schedules help balance fast convergence early with fine-tuning later.

A large learning rate makes big parameter updates, which is helpful early but harmful near a minimum because the steps overshoot and bounce around the optimum. This causes the loss to oscillate or plateau instead of settling. Reducing the rate later allows finer adjustments to reach a lower loss.

Step decay keeps the learning rate constant for a number of epochs and then multiplies it by a factor (such as 0.1) at fixed intervals. This creates a staircase pattern where the rate drops sharply at chosen milestones. It is simple and widely used in image classification training.

lr = initial_lr * (0.1 ** (epoch // 30))

In exponential decay the learning rate follows lr = lr0 * exp(-k*t), where lr0 is the initial rate and k is a positive decay constant. The rate decreases smoothly and continuously rather than in discrete steps. A larger k makes the rate drop faster.

import math
lr = lr0 * math.exp(-k * epoch)

Warmup means starting with a very small learning rate and gradually increasing it to the target value over the first few epochs or steps. It is used because large initial updates on randomly initialized weights can destabilize training, especially with large batch sizes or adaptive optimizers. After warmup, a normal decay schedule usually takes over.

Cosine annealing smoothly decreases the learning rate following a half cosine curve from the initial value down to a minimum (often near zero) over the training run. The rate falls slowly at first, faster in the middle, and slowly again near the end. It often gives better final accuracy than abrupt step drops because the decline is smooth.

import math
lr = lr_min + 0.5*(lr_max - lr_min)*(1 + math.cos(math.pi * t / T))

Warm restarts periodically reset the learning rate back to its maximum value and then anneal it down again, repeating in cycles. Each restart can help the optimizer escape a poor local region and explore new minima. This technique is sometimes combined with snapshot ensembling by saving the model at the end of each cycle.

Reduce-on-plateau monitors a metric such as validation loss and lowers the learning rate when that metric stops improving for a set number of epochs (the 'patience'). This makes the schedule adaptive to actual training progress rather than fixed in advance. It is useful when you do not know good milestones ahead of time.

scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(opt, patience=5, factor=0.1)

In linear decay the learning rate decreases by a constant amount each step from the initial value, reaching zero at the final step. So at the last step the learning rate is zero (or the specified end value). The formula is lr = lr0 * (1 - t/T) when decaying to zero over T steps.

lr = lr0 * (1 - t / T)

The 1cycle policy increases the learning rate from a low value to a high maximum during the first part of training, then decreases it back down (often below the start) during the second part, all within a single cycle. Momentum is usually varied inversely to the learning rate. It can enable faster training, sometimes called super-convergence.

scheduler = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=0.1, total_steps=1000)

An LR range test trains the model for a few iterations while increasing the learning rate from very small to large and records the loss at each rate. By plotting loss versus learning rate, you can pick a maximum rate just before the loss starts to diverge. It gives a data-driven way to choose a good learning rate instead of guessing.

The learning rate is the multiplier applied to the gradient to determine how far parameters move each update: new_weight = old_weight - lr * gradient. A larger learning rate produces bigger steps and a smaller one produces smaller steps. Schedules change this multiplier over time to control step size.

w = w - lr * grad

Smooth schedules avoid sudden large changes in step size, so training transitions gradually and tends to be more stable. Abrupt drops can cause temporary jumps in behavior, and the timing of step milestones must be tuned carefully. Cosine and similar curves remove the need to pick exact drop epochs and often reach slightly better final accuracy.

Polynomial decay reduces the learning rate following lr = lr0 * (1 - t/T)^p, where p is the power and T is the total steps. With p=1 it is linear decay; a higher p makes the rate fall faster at the start. It is common in training transformers and segmentation models.

lr = lr0 * (1 - t / T) ** p

Inverse time decay reduces the learning rate as lr = lr0 / (1 + k*t), so the rate shrinks roughly in proportion to one over the step number. The decrease is fast at first and then slows down. The parameter k controls how aggressively the rate decays.

lr = lr0 / (1 + k * t)

At epoch 65 the rate has been multiplied by 0.1 twice (at epochs 30 and 60), since 65 // 30 = 2. So lr = 0.1 * 0.1^2 = 0.1 * 0.01 = 0.001. The third drop would not occur until epoch 90.

lr = 0.1 * (0.1 ** (65 // 30))  # 0.001

Adaptive optimizers adjust an effective per-parameter step using gradient history, but they still multiply by a global base learning rate that you set. So you can and often do apply a schedule (such as warmup plus decay) to Adam's base rate on top of its adaptivity. Adaptivity and scheduling address different things and are commonly combined.

If the rate drops too fast, the model takes tiny steps before it has reached a good region, so it converges prematurely to a poor solution or trains very slowly. The early phase benefits from a relatively large rate to make rapid progress. Decaying too early wastes the benefit of fast exploration.

For epoch-based schedules such as StepLR you call scheduler.step() once at the end of each epoch, after the optimizer.step() calls for that epoch. This advances the schedule's internal epoch counter and updates the learning rate. Calling it at the wrong frequency (for example every batch) would make the rate decay far too fast.

for epoch in range(N):
    train_one_epoch()
    scheduler.step()

A triangular cyclical learning rate oscillates the rate linearly up and down between a lower and upper bound, forming a triangle wave repeated over cycles. The idea is that periodically increasing the rate helps the optimizer move out of saddle points and poor minima. The bounds are usually found with an LR range test.

Temporarily raising the learning rate gives the optimizer larger steps that can push it out of sharp or poor local minima and saddle points where it might otherwise get stuck. This added exploration can help it find flatter, more generalizable minima. The rate is then lowered again to refine the solution.

A common transformer schedule increases the learning rate linearly during a warmup period and then decreases it afterward, often proportional to the inverse square root of the step number. This 'warmup then inverse-sqrt decay' stabilizes early training and gradually fine-tunes later. The original Transformer paper used exactly this pattern.

lr = d_model**-0.5 * min(step**-0.5, step * warmup**-1.5)

Patience is the number of epochs with no improvement in the monitored metric that the scheduler waits before reducing the learning rate. A larger patience makes the scheduler more tolerant of temporary stalls, while a smaller patience reacts faster. Choosing it well prevents both premature and overly delayed reductions.

An epoch-based schedule changes the learning rate once per full pass over the dataset, while a step-based schedule changes it after each mini-batch update. Step-based scheduling gives finer, more frequent control and is common for warmup and cosine schedules in large-scale training. The total number of updates differs, so the schedule math must match the granularity used.

At t=T/2 the cosine term cos(pi*0.5)=0, so lr = lr_min + 0.5*(lr_max - lr_min)*(1+0) = 0.5*0.1 = 0.05. Thus the rate is exactly halfway between the maximum and minimum at the midpoint. This reflects the symmetric shape of the cosine curve.

import math
lr = 0 + 0.5*(0.1-0)*(1+math.cos(math.pi*0.5))  # 0.05

Large batches give more accurate gradient estimates and often need a higher peak learning rate to train efficiently, but jumping straight to that high rate can destabilize early training. Warmup schedules let the rate ramp up safely, and decay later helps convergence. This is why large-batch training pipelines almost always include carefully designed schedules.

If the learning rate is too high, the training loss may increase, oscillate wildly, or become NaN instead of steadily decreasing. These signs indicate the updates are overshooting. Lowering the rate or adding warmup usually stabilizes training.

The factor (often called gamma) is the number the current learning rate is multiplied by when a reduction happens, such as 0.1 or 0.5. A factor of 0.1 cuts the rate to one tenth at each drop. Smaller factors cause more aggressive reductions.

scheduler = torch.optim.lr_scheduler.StepLR(opt, step_size=30, gamma=0.1)

By lowering the rate near the end of training, schedules let the optimizer settle into wider, flatter minima that tend to generalize better to unseen data. Some cyclical and warm-restart schedules also explore multiple minima, which can be averaged into ensembles. So schedules can improve test accuracy beyond just speeding up convergence.

During linear warmup the rate is lr = target * (step / warmup_steps) = 0.001 * (40/100) = 0.0004. So at step 40 the learning rate is 0.0004. It will keep rising until it reaches 0.001 at step 100.

lr = 0.001 * (40 / 100)  # 0.0004

Snapshot ensembling trains a single model with a cyclical (often cosine warm-restart) schedule and saves a copy of the model at the end of each cycle, when the rate is lowest. These snapshots represent different good minima and are combined like an ensemble at test time. It produces ensemble-like accuracy gains at the cost of training only one model.

The learning rate is a hyperparameter: it is set by the practitioner or a schedule, not learned directly by gradient descent like the weights. A schedule is a predetermined or metric-driven rule for changing this hyperparameter over time. So scheduling automates how the hyperparameter evolves rather than learning it from gradients.

A very large k makes exp(-k*t) shrink extremely fast, so the learning rate collapses toward zero after only a few steps. The model then effectively stops learning very early, likely underfitting. Choosing k too large therefore halts useful training prematurely.

import math
lr = lr0 * math.exp(-k * t)  # large k => tiny lr quickly

Compare them on a held-out validation or test set after the same total training budget, since the goal is good generalization, not just low training loss. You should also account for randomness by averaging over a few seeds if possible. Judging only by final training loss can be misleading because a schedule might overfit.