Gradient descent and optimization

Gradient descent is an iterative optimization algorithm that minimizes a function by repeatedly moving the parameters in the direction opposite to the gradient (the direction of steepest decrease). Each step reduces the loss until it reaches (or approaches) a minimum.

α is the learning rate, which controls the size of each step taken along the negative gradient. If α is too large the algorithm can overshoot and diverge, and if it is too small convergence becomes very slow.

The gradient points in the direction of steepest increase of the function. To minimize, we want to decrease the value, so we move in the opposite direction, which is the direction of steepest decrease.

Batch gradient descent computes the gradient using the entire training dataset for each update, giving a stable but slow step. SGD uses a single randomly chosen example per update, making it much faster and noisier, which can help escape shallow local minima.

Mini-batch gradient descent computes the gradient on a small subset (e.g., 32 or 64 samples) of the data per update. It balances the stability of batch descent with the speed and noise benefits of SGD, and it makes efficient use of vectorized hardware like GPUs.

A local minimum is a point where the function is lower than all nearby points but not necessarily the lowest overall (global minimum). Gradient descent can get stuck there because the gradient is zero, so the algorithm stops even though a better solution exists elsewhere.

A convex function is bowl-shaped: any line segment between two points on the curve lies on or above the curve, so it has a single global minimum and no other local minima. Because there are no traps, gradient descent with a suitable learning rate will converge to that unique global minimum.

A learning rate that is too high causes the parameters to overshoot the minimum on each step, so the loss may oscillate, increase, or diverge to infinity instead of decreasing. The training curve often shows the loss bouncing around or blowing up.

Momentum adds a fraction of the previous update to the current one, like a ball rolling downhill accumulating velocity. It helps accelerate descent in consistent directions and dampens oscillations in narrow ravines, speeding up convergence.

Adam combines momentum with per-parameter adaptive learning rates, using running averages of past gradients and their squares. This makes it converge quickly and work well across many problems with little manual tuning of the learning rate.

An epoch is one complete pass of the optimization algorithm through the entire training dataset. Training usually requires many epochs so the model sees the data repeatedly and gradually reduces the loss.

The update is x_new = x − α·f'(x) = 4 − 0.1·(2·4) = 4 − 0.8 = 3.2. The point moves closer to the minimum at x = 0, as expected.

x = 4.0
alpha = 0.1
grad = 2 * x      # f'(x) = 2x
x = x - alpha * grad
print(x)          # 3.2

Convergence means the parameter updates become very small and the loss stops decreasing meaningfully, indicating the algorithm has reached (or is very near) a minimum. In practice we detect it when the loss change falls below a small threshold or the gradient is near zero.

When features have very different scales, the loss surface becomes elongated, causing gradient descent to zigzag and converge slowly. Scaling features to comparable ranges makes the surface more symmetric, allowing larger, more direct steps toward the minimum.

A saddle point is where the gradient is zero but the function increases in some directions and decreases in others, so it is neither a maximum nor a minimum. In high dimensions saddle points are very common and can slow gradient descent because the gradient is small near them.

In deep networks, gradients are multiplied through many layers during backpropagation, and if these values are small they shrink exponentially, becoming nearly zero in early layers. This means early layers barely update, making training very slow or stalled.

A learning rate schedule gradually reduces the learning rate as training progresses. Large early steps move quickly toward the minimum, while smaller later steps allow fine-tuning and stable settling into the minimum without overshooting.

The cost function measures how far the model's predictions are from the true values, producing a single number to minimize. Gradient descent uses the gradient of this cost function with respect to the parameters to decide which direction to update them.

SGD estimates the gradient from a single example (or small batch), so each estimate differs from the true full-dataset gradient, producing noisy zigzag updates. This noise can be beneficial because it helps the optimizer escape shallow local minima and saddle points.

Exploding gradients occur when gradients grow extremely large during backpropagation, causing huge unstable parameter updates and loss that becomes NaN or infinite. A common fix is gradient clipping, which caps the gradient magnitude at a maximum threshold.

At a minimum the function is locally flat, so its slope (derivative) is zero, meaning no direction of small movement decreases the value further. Gradient descent naturally stops there because the update step θ − α·∇J becomes θ when the gradient is zero.

A global minimum is the lowest value the function takes over its entire domain, while a local minimum is only the lowest within some neighborhood. A function can have many local minima but the global minimum is the absolute best point.

`grad` is the derivative of the loss with respect to w, which is 2*(w-3); it tells the direction and steepness for the update. Subtracting `lr*grad` moves w toward 3, the value that minimizes the loss.

w = 0.0
lr = 0.1
for _ in range(50):
    grad = 2 * (w - 3)   # d/dw of (w-3)**2
    w = w - lr * grad
print(round(w, 3))       # approaches 3.0

Early stopping halts training when performance on a validation set stops improving, even if the training loss could keep decreasing. It prevents the optimizer from over-minimizing the training loss to the point of overfitting, acting as a form of regularization.

In a ravine the surface is steep across the narrow direction but gently sloped along the long direction, so gradients point mostly across, causing the path to zigzag back and forth. This wastes steps and makes progress along the true descent direction slow; momentum or adaptive methods help.

Batch size is the number of training examples used to compute the gradient for each update. Very large batches give smoother, more accurate gradient estimates but require more memory and may generalize worse and provide fewer updates per epoch.

Backpropagation computes the gradient of the loss with respect to each weight by applying the chain rule, multiplying derivatives layer by layer from the output back to the input. This efficiently propagates error signals so every parameter knows how to change to reduce the loss.

The magnitude of the gradient indicates how steep the slope is, so larger gradients produce bigger steps and smaller gradients produce smaller steps for a fixed learning rate. This naturally slows the algorithm down as it approaches a flat minimum where the gradient shrinks.

RMSProp scales the learning rate for each parameter by a running (exponentially decayed) average of recent squared gradients. This fixes AdaGrad's issue where the accumulated squared gradients grow without bound and shrink the learning rate to nearly zero, stalling learning.

A sudden upward spike usually indicates an unstable step, often from a learning rate that is too high causing overshooting or from exploding gradients. Reducing the learning rate or applying gradient clipping commonly fixes it.

Gradient descent relies on computing the gradient (derivatives) of the loss with respect to the parameters to know which way to step. If the function is not differentiable, the gradient may not exist at some points, so the algorithm cannot determine a valid descent direction there.

Parameters (weights and biases) are values the model learns automatically through gradient descent during training. Hyperparameters like the learning rate are set by the practitioner before training and control how the learning process behaves, not the model's internal values.

The starting point determines where on the loss surface descent begins, affecting which minimum is reached and how fast. Poor initialization (e.g., all zeros or very large values) can cause symmetry problems, vanishing or exploding gradients, and slow or failed convergence.