What is gradient descent in simple terms?

An algorithm that improves a model step by step: measure how wrong the model is (the loss), work out which direction reduces that error fastest (the negative gradient), take a small step that way, and repeat until the error stops improving.

What happens if the learning rate is too high or too low?

Too high and updates overshoot — the loss oscillates or explodes. Too low and training is painfully slow and can stall on plateaus. Practitioners tune it, often with warmup and decay schedules.

What is the difference between batch and stochastic gradient descent?

Batch computes the gradient over the whole dataset per update — exact but slow. Stochastic uses one example per update — fast but noisy. Mini-batch, the modern default, uses small batches to get the best of both.

How does backpropagation relate to gradient descent?

Backpropagation is the algorithm that efficiently computes the gradient of the loss through a neural network's layers. Gradient descent then uses that gradient to update the weights. Backprop finds the slope; gradient descent takes the step.

GuideJuly 15, 2026Learn AI & Data Science

Gradient Descent Explained Simply: How Machines Actually Learn

Gradient descent explained with the downhill-walk intuition: loss functions, gradients, learning rates, and the batch/mini-batch/stochastic variants.

Strip away the mathematics and gradient descent is a hiker descending a foggy mountain. You can't see the valley (the best possible model), but you can feel the slope under your feet. So you take a step in the steepest downhill direction, feel again, and step again. Repeat enough times and you end up somewhere low. That's the entire algorithm that trains everything from linear regression to the largest neural networks — the rest is detail about how big the steps are and how the slope is measured.

The three ingredients

A loss function — a single number measuring how wrong the model currently is (mean squared error for regression, cross-entropy for classification). The 'mountain' is this loss plotted over all possible parameter values.
The gradient — the vector of partial derivatives of the loss with respect to each parameter. It points in the direction of steepest *increase*, so we step the opposite way. In neural networks, backpropagation is simply the efficient way to compute this gradient.
The learning rate — the step size. Each update is: parameter ← parameter − learning rate × gradient.

The learning rate is the whole game

Too small a step and training crawls — thousands of epochs to inch downhill. Too large and you overshoot the valley, bouncing across it or diverging entirely as the loss climbs. Getting this dial right matters so much that it has its own tricks — schedules, warmup, adaptive methods — which we unpack in learning rate explained.

Batch, stochastic, and mini-batch

Variant	Gradient computed on	Character
Batch gradient descent	The entire dataset per step	Exact but slow and memory-hungry; impractical for large data
Stochastic (SGD)	One example per step	Fast, noisy steps; the noise can help escape poor local minima
Mini-batch (the default)	A small batch, e.g. 32–512 examples	The practical sweet spot — stable enough, fast on GPUs

Doesn't it get stuck in local minima?

In low dimensions, it can. In the very high-dimensional loss surfaces of deep networks, research suggests the more common obstacles are saddle points and flat plateaus rather than bad local minima — and the noise in mini-batch updates helps the optimizer move through them.

Modern training rarely uses vanilla gradient descent: momentum, RMSProp and Adam adapt the step per-parameter and are covered on the gradient descent variants (SGD, Adam) topic page, with the calculus itself in the math of gradient descent. To make it stick, **AI Learning** gives you illustrated guides plus ~10,000 practice questions — including optimization questions from intuition level up to interview depth — fully offline, no account.

AI Learning: AI/ML/DS Q&A

Free · Works offline · From the downhill intuition to interview-grade optimization questions.

Coming soon toGoogle Play

Sources