Gradient descent variants (SGD, Adam)

Gradient descent minimizes a loss function by repeatedly moving the parameters in the direction opposite to the gradient of the loss. The gradient points toward the steepest increase, so stepping the other way decreases the loss. The size of each step is controlled by the learning rate.

Batch Gradient Descent computes the gradient using the entire training dataset before each parameter update, giving smooth but slow updates. SGD updates the parameters using one training example at a time, making updates fast and noisy. The noise can actually help escape shallow local minima.

A mini-batch is a small subset of the training data (e.g., 32 or 64 examples) used to compute each gradient update. It balances the stability of batch gradient descent with the speed and noise benefits of SGD. Mini-batches also make efficient use of vectorized hardware like GPUs.

The learning rate is a hyperparameter that scales how large a step is taken in the direction opposite the gradient. A learning rate that is too large can cause the loss to overshoot or diverge, while one too small makes training painfully slow. Choosing a good learning rate is one of the most important tuning decisions.

Because SGD estimates the gradient from a single example (or small mini-batch), the estimate fluctuates around the true full-dataset gradient. This causes the loss to bounce up and down rather than decrease smoothly. The noise is a tradeoff: it can slow convergence near the minimum but helps explore the loss surface.

Momentum addresses slow progress and oscillation, especially in ravine-like regions where the loss curves steeply in one direction and gently in another. It accumulates a running average of past gradients, like a ball rolling downhill gaining speed, so consistent directions are amplified and oscillations are damped. This usually speeds up convergence.

β controls how much weight is given to past gradients versus the current gradient g. A higher β (e.g., 0.9) means the velocity v retains more history and changes direction slowly, giving stronger smoothing. A β of 0 reduces the update back to plain gradient descent. (Note: this is the exponential moving average form; some libraries like PyTorch instead use v = β·v + g.)

AdaGrad gives each parameter its own effective learning rate that adapts based on the history of its gradients. Parameters with large accumulated squared gradients get smaller effective learning rates, while infrequently updated parameters keep larger ones. This is helpful for sparse features, but the accumulated sum keeps growing and can shrink the step size until learning stalls.

AdaGrad accumulates all past squared gradients, so its denominator keeps growing and the effective learning rate shrinks toward zero, halting learning. RMSProp fixes this by using an exponentially decaying moving average of squared gradients instead of a cumulative sum. This keeps the effective learning rate from vanishing, making it suitable for non-stationary problems.

Adam combines momentum and RMSProp. From momentum it keeps a moving average of the gradients (the first moment) to give direction and speed, and from RMSProp it keeps a moving average of squared gradients (the second moment) to adapt the per-parameter step size. Together these give fast, stable, adaptive updates.

The first moment m is an exponentially weighted average of the gradients, estimating their mean (direction and magnitude). The second moment v is an exponentially weighted average of the squared gradients, estimating their uncentered variance (how large/volatile they are). Adam divides m by the square root of v to scale each parameter's step.

Adam initializes the first and second moment estimates at zero, which biases them toward zero during the early steps of training. Bias correction divides each estimate by (1 - β^t), where t is the timestep, to compensate for this initialization bias. Without it, the early updates would be too small.

The commonly used defaults are β1 = 0.9 (for the first moment), β2 = 0.999 (for the second moment), and ε = 1e-8 (a small constant for numerical stability). These defaults work well across many problems, which is part of why Adam is so popular. The learning rate is typically set around 0.001.

Epsilon is a tiny positive number added to the denominator to prevent division by zero when the moving average of squared gradients is very small. It ensures numerical stability so the update does not blow up to infinity. A typical value is 1e-8.

Adam adapts the learning rate per parameter and incorporates momentum, so it usually converges faster and is less sensitive to the initial learning rate choice. This makes it a robust, low-tuning default that works well across many architectures. However, well-tuned SGD with momentum can sometimes generalize better.

With a learning rate that is too high, the parameter steps overshoot the minimum and the loss can oscillate wildly or diverge to infinity (NaN). Instead of settling into a low-loss region, the optimizer bounces around or escapes entirely. This is a common cause of training instability.

A learning rate schedule changes the learning rate over the course of training, often starting high and decreasing over time (decay). A large initial rate makes fast progress, while a smaller later rate allows fine-tuning near the minimum without overshooting. Common schedules include step decay, exponential decay, and cosine annealing.

The update is θ = θ - η·g, where θ is the parameter vector, η is the learning rate, and g is the gradient of the loss with respect to θ for the current example or mini-batch. The minus sign moves the parameters opposite the gradient to reduce the loss.

theta = theta - learning_rate * grad

Larger mini-batches give smoother, more accurate gradient estimates and use hardware (GPUs) more efficiently, but each step costs more memory and computation. Smaller batches add noise that can help generalization and require less memory, but are less hardware-efficient. The batch size is a tradeoff between stability, speed, and generalization.

An epoch is one complete pass through the entire training dataset. If the data is split into mini-batches, one epoch consists of as many update steps as there are batches. Models are usually trained for many epochs until the loss converges or validation performance stops improving.

A saddle point is a location where the gradient is zero but it is a minimum in some directions and a maximum in others, so it is not a true minimum. In high-dimensional deep networks, saddle points are far more common than bad local minima and can dramatically slow plain gradient descent. Momentum and adaptive methods help push through them.

Because momentum accumulates velocity from previous gradients, it keeps moving even when the current gradient is small or briefly points the wrong way. This built-up speed carries the parameters across flat plateaus and over tiny local bumps. It behaves like inertia, smoothing the trajectory toward the minimum.

Classical momentum computes the gradient at the current position and then combines it with the accumulated velocity. Nesterov momentum first takes a 'lookahead' step in the direction of the velocity and computes the gradient there, which lets it anticipate and correct course sooner. This lookahead often gives slightly faster and more stable convergence.

You create the optimizer with torch.optim.Adam, passing the model parameters and a learning rate, then in each step you zero the gradients, compute the loss, backpropagate, and call optimizer.step(). The optimizer object internally maintains the moment estimates for Adam. Forgetting optimizer.zero_grad() causes gradients to accumulate incorrectly.

import torch
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
opt.zero_grad()
loss = loss_fn(model(x), y)
loss.backward()
opt.step()

By default, frameworks like PyTorch accumulate gradients by adding new gradients to existing ones. If you do not zero them out each step, gradients from previous batches pile up and corrupt the current update. Zeroing the gradients ensures each step uses only the current batch's gradient.

opt.zero_grad()  # reset before backward()

Gradient clipping caps the magnitude (norm) of gradients to a maximum threshold before the update step. It addresses the exploding gradient problem, where very large gradients (common in deep or recurrent networks) cause unstable, divergent updates. By limiting the gradient size, training stays stable.

torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

Adam's adaptive per-parameter learning rates can converge to sharp minima that fit the training data tightly but generalize less well to new data. Plain SGD with momentum tends to settle in flatter, wider minima that often generalize better. This is why some practitioners switch from Adam to SGD late in training or use AdamW.

In standard Adam, L2 regularization gets entangled with the adaptive learning rate, so weight decay is scaled differently for each parameter and is less effective. AdamW decouples weight decay from the gradient-based update, applying it directly to the weights. This 'decoupled weight decay' often improves generalization.

A sudden jump to NaN usually means the gradients or updates became numerically unstable, often due to a learning rate that is too high causing the loss to explode. Exploding gradients or numerical overflow in the loss can also produce NaNs. Common fixes are lowering the learning rate, adding gradient clipping, or checking the data for bad values.

During warmup, the learning rate starts very small and gradually increases to its target value over the first few hundred or thousand steps. Early in training the gradients and Adam's moment estimates are unreliable, so small steps prevent instability. Warmup is especially common when training large models like Transformers.

As β approaches 1, the optimizer averages over many more past gradients, so it builds up larger velocity and responds more slowly to changes in the gradient direction. This gives strong smoothing and acceleration in consistent directions but can overshoot if too high. Values like 0.9 to 0.99 are typical sweet spots.

It tells you how much the loss would change for a small increase in that parameter, including the direction (sign) and rate (magnitude) of change. A positive gradient means increasing the parameter increases the loss, so you should decrease it. Gradient descent uses this information for every parameter at once to lower the loss.

Adam scales the step for each parameter by dividing by the running magnitude of its gradients, so parameters with small or rare gradients still get meaningful updates and those with large gradients are tamed. Combined with momentum, this lets it take well-sized steps in every direction from the start. Vanilla SGD uses a single global learning rate, so it cannot adapt this way.

It typically indicates the learning rate is too high, so each step overshoots and the optimizer cannot settle into a lower-loss region. The remedy is usually to reduce the learning rate or add a decay schedule. Persistent oscillation with no downward trend means the steps are too aggressive for the current loss landscape.