Batch vs stochastic training

Batch gradient descent computes the gradient using the entire training dataset before making a single parameter update. Stochastic gradient descent uses just one randomly chosen training example to compute the gradient and update parameters each step. This means SGD makes many more, noisier updates per pass through the data.

'Stochastic' means involving randomness. In SGD, the randomness comes from selecting a single training example (or shuffling the data) to estimate the gradient, rather than using the full dataset. This makes each gradient a noisy, random estimate of the true full-dataset gradient.

Pure SGD makes N updates in one epoch (one update per example), while batch gradient descent makes only 1 update per epoch (after seeing all N examples). So SGD performs far more frequent updates per epoch.

Mini-batch gradient descent computes the gradient using a small subset (e.g., 32 or 64 examples) of the data per update. It is a compromise between full-batch (all examples) and pure SGD (one example), capturing benefits of both: more stable than SGD and more frequent updates than batch.

Because each update is based on a single example, the gradient estimate varies a lot from step to step and only approximates the true gradient direction. This variance causes the parameter trajectory to fluctuate rather than move smoothly, producing a zig-zag path toward the minimum.

Stochastic (and mini-batch) gradient descent is more memory-efficient because it only needs one or a few examples in memory at a time. Batch gradient descent requires loading or processing the entire dataset to compute each gradient, which can be infeasible for very large datasets.

The random fluctuations in SGD gradients can push the parameters out of shallow local minima or away from saddle points where a smooth batch method might get stuck. This noise acts like a small random kick, helping the optimizer explore and potentially find better minima.

Even at the true minimum, individual examples have nonzero gradients, so a fixed-step SGD keeps taking steps and bounces around the optimum. To converge precisely, the learning rate must be decreased over time so the steps shrink as the minimum is approached.

An epoch is one complete pass through the entire training dataset. In mini-batch training, an epoch consists of multiple updates, one per mini-batch, until all examples have been used once. Training typically runs for many epochs.

The number of updates per epoch equals the dataset size divided by the batch size: 10,000 / 100 = 100 updates. Each mini-batch produces one update, so there are 100 updates in a single epoch.

GPUs perform parallel matrix operations very efficiently, so processing a mini-batch of examples at once exploits this parallelism and is much faster per example than processing one example at a time. Pure SGD underuses the hardware because each step involves only one example.

Larger mini-batches average over more examples, which reduces the variance (noise) of the gradient estimate, making updates smoother and closer to the true full-batch gradient. Smaller batches give noisier but more frequent updates.

False. Batch gradient descent guarantees convergence to the global minimum only for convex loss functions. For non-convex losses (typical in deep neural networks), it can get stuck in local minima or saddle points, just like other gradient methods.

One SGD step is much cheaper because it computes the gradient from a single example, whereas one batch step processes the entire dataset. However, SGD needs many more steps, so total work per epoch is comparable.

Shuffling prevents the model from learning the order of examples and reduces the risk of the optimizer following a biased, repetitive update pattern. It ensures mini-batches are roughly representative random samples each epoch, improving the quality and stability of training.

For SGD, the update uses a single example's gradient: w := w - lr * 2 * (w*xi - yi) * xi. Batch GD instead averages this gradient over all N examples before updating. The structural difference is summing/averaging over all data (batch) versus using one term (SGD).

# SGD update for one example (linear model y = w*x, MSE)
grad = 2 * (w * xi - yi) * xi
w = w - lr * grad

Very large batches make updates smooth and accurate but reduce the number of updates per epoch and require more memory. They can also converge to 'sharp' minima that generalize worse, and they lose the helpful regularizing noise that smaller batches provide.

With SGD or mini-batches, you get a parameter update and loss signal after each small step, giving rapid feedback within an epoch. With full-batch GD, you only update once per epoch, so progress feedback is much slower, especially on large datasets.

A decreasing (decaying) learning rate lets SGD take large exploratory steps early and progressively smaller steps later, reducing the oscillation caused by gradient noise near the minimum. This helps the noisy SGD process actually converge close to the optimum instead of bouncing around forever.

Yes, if the mini-batch is sampled uniformly at random, its average gradient is an unbiased estimate of the full-dataset gradient, meaning its expected value equals the true gradient. The mini-batch gradient has higher variance but no systematic bias.

Full-batch gradient descent follows the smoothest, most direct path because it uses the exact gradient of the whole dataset each step. SGD and small mini-batches zig-zag due to noisy, approximate gradients.

Batch GD needs the entire dataset to compute each gradient, so if the data exceeds available memory, you cannot process it all at once. Mini-batch or stochastic methods solve this by streaming small chunks, computing gradients on subsets that do fit in memory.

Online learning updates the model as each new data point arrives, one at a time, rather than retraining on the full dataset. It closely resembles stochastic gradient descent, since both update parameters using a single example at a time.

Smaller mini-batches introduce gradient noise that prevents the model from fitting the training data too precisely, similar to adding randomness that improves generalization. This noise can help the model find flatter minima that perform better on unseen data, acting as implicit regularization.

This implements stochastic gradient descent because it loops over individual examples and updates the weights w using one example's gradient at a time. A batch version would compute the gradient over all examples first, then make a single update outside the loop.

for i in range(N):
    grad = compute_grad(X[i], y[i])
    w -= lr * grad

Because the gradients differ in magnitude and noise: batch gradients are smooth and averaged, allowing larger stable steps, while SGD gradients are noisy and can overshoot, often requiring a smaller or decaying learning rate. The right learning rate depends on the batch size and gradient variance.

The total number of example gradients computed per epoch stays roughly the same (equal to N, the dataset size) regardless of batch size, since every example is seen once. What changes is how those computations are grouped: small batches mean many updates, large batches mean few updates.

Because SGD updates parameters after each example, the model begins improving immediately from the first few examples, rather than waiting to process all data. This early progress makes SGD especially effective when datasets are huge and a single full pass is expensive.

On GPUs and modern CPUs, processing 256 examples in parallel as one batch is usually faster in wall-clock time than 256 separate single-example steps, because parallel hardware processes a batch almost as quickly as one example. Batch size 1 underuses the hardware, so it is often slower overall despite cheaper individual updates.

The standard deviation of the mini-batch gradient noise decreases approximately as 1/sqrt(m), where m is the batch size. So to halve the noise you need about four times as many examples per batch, showing diminishing returns from very large batches.

Full-batch GD always uses the entire dataset, so the gradient and resulting update are the same every run given the same starting point, making it deterministic. SGD depends on the random order or random selection of examples, so different random seeds produce different update sequences and slightly different results.

Mini-batch training balances the strengths of both extremes: it is computationally efficient on parallel hardware, uses manageable memory, gives frequent updates with helpful but controlled noise, and tends to generalize well. This makes it the practical default for training neural networks.

You often need to lower the learning rate (or use a schedule) because smaller batches give noisier gradients that can cause unstable, overshooting updates at a high learning rate. A common heuristic is to scale the learning rate roughly with the batch size, increasing it for larger batches.

Full-batch GD usually produces a smooth, steadily decreasing loss curve because each update uses the exact gradient. SGD produces a jagged, fluctuating loss curve because each update is based on a single noisy example, even though the overall trend still decreases.