The math of gradient descent

Gradient descent updates a parameter by moving it opposite to the gradient of the loss: w := w − α·(dL/dw). The gradient points in the direction of steepest increase, so subtracting it decreases the loss. The learning rate α controls how big each step is.

The gradient points in the direction in which the loss increases fastest. To minimize the loss we want to move in the opposite direction, so we subtract the gradient. Adding it would perform gradient ascent and increase the loss.

The derivative of w² is 2w, so dL/dw = 2·3 = 6. The gradient is positive, meaning the loss increases as w increases at this point, so gradient descent would decrease w.

The learning rate scales the size of each update step. A small α gives slow but stable convergence, while a large α can speed things up but risks overshooting or diverging. Choosing a good α is one of the most important tuning decisions.

With too large a learning rate the steps overshoot the minimum, and the loss can oscillate or even grow without bound (diverge). Instead of settling into the valley, the parameter bounces back and forth across it. This is why learning rates are often kept small.

The gradient is the vector of partial derivatives of L with respect to each parameter. It points in the direction of steepest ascent of the loss surface, and its magnitude indicates how steep that ascent is. Gradient descent moves opposite to this vector.

The gradient is 2w = 10 at w = 5. The update is w := 5 − 0.1·10 = 5 − 1 = 4. So after one step w becomes 4, moving closer to the minimum at 0.

w = 5
for _ in range(1):
    grad = 2 * w
    w = w - 0.1 * grad
print(w)  # 4.0

Batch gradient descent computes the gradient using the entire training dataset before each update, giving a smooth but expensive step. SGD updates parameters using just one (or a few) randomly chosen examples at a time, making it faster and noisier. The noise can help escape shallow local minima.

At a local or global minimum of a smooth function, the gradient is zero because the slope is flat there. This is why gradient descent slows down and effectively stops as it approaches a minimum. A zero gradient is a necessary (but not sufficient) condition for a minimum.

A mini-batch is a small subset of the training data (e.g., 32 or 64 examples) used to compute each gradient update. Mini-batch gradient descent balances the stability of batch descent with the speed and noise of SGD, and it makes efficient use of vectorized hardware like GPUs. It is the standard approach in deep learning.

An epoch is one complete pass through the entire training dataset. With mini-batches, one epoch consists of many update steps, one per mini-batch. Training usually involves many epochs so the model sees the data repeatedly.

The minimum is at w = 4, where the squared term equals zero. The derivative is 2(w − 4), which evaluates to 0 at w = 4. So gradient descent converges toward w = 4.

Neural networks are compositions of many functions, and the chain rule lets us compute the derivative of the loss with respect to each weight by multiplying derivatives layer by layer. This is exactly what backpropagation does. Without the chain rule, computing gradients through deep networks would be intractable.

A convex function curves upward so that any line segment between two points on its graph lies on or above the curve, giving it a single global minimum. For convex losses, gradient descent (with a suitable learning rate) is guaranteed to converge to that global minimum. Non-convex losses may have many local minima where descent can get stuck.

Vanishing gradients occur when gradients become extremely small as they are propagated backward through many layers, so early-layer weights barely update. This slows or stalls learning in deep networks. Activation functions like ReLU and techniques like residual connections help mitigate it.

Momentum accumulates an exponentially weighted moving average of past gradients and uses it to update parameters, like a ball rolling downhill gaining speed. This dampens oscillations across steep directions and accelerates progress along consistent directions. It often leads to faster, smoother convergence.

The partial derivatives are ∂L/∂w1 = 2w1 and ∂L/∂w2 = 2w2. At (1, 2) this gives (2·1, 2·2) = (2, 4). Gradient descent would move opposite to (2, 4).

Convergence means the parameter updates become very small and the loss stops decreasing meaningfully, indicating the algorithm has reached (or settled near) a minimum. In practice we detect this when the change in loss falls below a small threshold. It does not always mean the global minimum has been found.

Differentiating the squared term (yi − ŷi)² with respect to ŷi gives −2(yi − ŷi) = 2(ŷi − yi). So each prediction's gradient is proportional to how far it is from the true value. Larger errors produce larger gradients and therefore larger corrections.

When features have very different ranges, the loss surface becomes elongated, causing gradient descent to zig-zag and converge slowly. Scaling features to similar ranges makes the surface more spherical so descent takes a more direct path. This typically speeds up convergence significantly.

A learning rate schedule reduces the learning rate over time during training, for example by lowering it after a fixed number of epochs. Large early steps make fast progress, while smaller later steps allow fine-tuning near the minimum without overshooting. This often improves final accuracy.

A very small learning rate makes each step tiny, so training takes an extremely long time to reach the minimum. It can also appear to make almost no progress and may stall in shallow regions of the loss surface. It is computationally wasteful even though it is stable.

Each step uses gradient 2w: from 10 → 10−0.1·20=8, then 8→8−0.1·16=6.4, then 6.4→6.4−0.1·12.8=5.12. So after 3 steps w = 5.12, approaching the minimum at 0.

w = 10.0
for _ in range(3):
    grad = 2 * w
    w = w - 0.1 * grad
print(w)  # 5.12

A saddle point is where the gradient is zero but the surface curves up in some directions and down in others, so it is neither a minimum nor a maximum. Because the gradient is near zero there, plain gradient descent can slow down or stall around saddle points. Momentum-based methods help push through them.

Adam combines momentum (a running average of gradients) with per-parameter adaptive learning rates (a running average of squared gradients). This lets each parameter take appropriately sized steps and smooths the trajectory. It often converges faster and is less sensitive to the initial learning rate than plain SGD.

The magnitude of the gradient measures how steep the loss surface is at the current point. A large norm means a steep slope and large potential update, while a norm near zero suggests you are close to a flat region or minimum. Monitoring it helps detect convergence or exploding gradients.

Gradient clipping limits the magnitude of gradients to a maximum threshold before the update, rescaling them if they exceed it. It is used to prevent exploding gradients, especially in recurrent networks, which would otherwise cause huge, destabilizing updates. It keeps training stable without changing the gradient's direction.

The derivative is 2w + 4, which equals zero at w = −2. Since the function (w+2)² is convex with minimum at w = −2, gradient descent converges there. The minimum loss value is 0.

Gradient descent relies on computing derivatives of the loss to know which direction reduces it, so the loss must be differentiable to provide a gradient. Where it is not differentiable, the gradient is undefined and the standard update cannot be computed. Some functions like ReLU use subgradients to handle isolated non-differentiable points.

Because SGD estimates the gradient from a small random sample, its updates are noisy rather than exact. This noise can help the optimizer jump out of shallow local minima or saddle points and explore the loss surface. It also makes each update much cheaper to compute.

Since the update is w := w − α·(∂L/∂w) and the derivative is negative, subtracting a negative number increases w. So gradient descent moves w in the positive direction, toward lower loss. The rule always moves opposite to the sign of the slope.

The loss surface is a multidimensional landscape where each axis is a model parameter and the height is the loss value for that combination of parameters. Gradient descent navigates this surface by repeatedly stepping downhill toward lower-loss regions. Training is essentially the search for the lowest valley on this surface.

Non-convex losses have multiple local minima and saddle points, so the path downhill depends on where you start. Different initial parameters can lead descent into different valleys, giving different final solutions. This is why weight initialization and running multiple seeds can matter in deep learning.

At each step you need the gradient ∇L of the loss with respect to the parameters (the direction) and you apply the chosen learning rate α (the step size). Multiplying them gives the update amount, which is subtracted from the current parameters. The learning rate is typically fixed or scheduled, while the gradient is recomputed each step.