Batch vs stochastic training quiz

Question 1

What is the key difference between batch gradient descent and stochastic gradient descent (SGD) in terms of how many training examples are used per parameter update?

Accepted Answer

Batch uses the whole dataset per update; SGD uses one example per update

Answer

Batch uses one example per update; SGD uses the whole dataset

Answer

Both use the whole dataset, but SGD is faster

Answer

Both use one example, but batch repeats it many times

Question 2

What does 'stochastic' mean in the context of stochastic gradient descent?

Accepted Answer

Involving randomness

Answer

Deterministic and exact

Answer

Using a fixed learning rate

Answer

Without any gradient

Question 3

In one full pass over a dataset of N examples, roughly how many parameter updates does pure SGD make compared to batch gradient descent?

Accepted Answer

SGD: N updates, Batch: 1 update

Answer

SGD: 1 update, Batch: N updates

Answer

Both make N updates

Answer

Both make 1 update

Question 4

What is mini-batch gradient descent, and how does it relate to batch and stochastic methods?

Accepted Answer

It uses a small subset of examples per update

Answer

It uses all examples at once

Answer

It uses exactly one example per update

Answer

It avoids gradients entirely

Question 5

Why does stochastic gradient descent produce a 'noisy' or zig-zagging path toward the minimum?

Accepted Answer

Because each single-example gradient is a noisy estimate of the true gradient

Answer

Because it uses too small a learning rate

Answer

Because it never reaches the minimum

Answer

Because the dataset is too large

Question 6

Which method, batch or stochastic gradient descent, is generally more memory-efficient for very large datasets, and why?

Accepted Answer

Stochastic, because it processes few examples at a time

Answer

Batch, because it is simpler

Answer

Both use the same memory

Answer

Neither can handle large data

Question 7

How can the noise in SGD updates actually help training escape poor local minima or saddle points?

Accepted Answer

Noise can help escape shallow local minima and saddle points

Answer

Noise always prevents convergence

Answer

Noise has no effect on minima

Answer

Noise guarantees the global minimum

Question 8

For a convex loss function with a fixed learning rate, why might pure SGD oscillate around the minimum instead of settling exactly on it?

Accepted Answer

Because single-example gradients are nonzero even at the minimum, with a fixed step

Answer

Because the loss is non-convex

Answer

Because batch gradient is always zero

Answer

Because SGD ignores the loss function

Question 9

What is an 'epoch' in the context of mini-batch training?

Accepted Answer

One complete pass through the training dataset

Answer

One parameter update

Answer

One mini-batch

Answer

One single example

Question 10

If a dataset has 10,000 examples and the mini-batch size is 100, how many parameter updates occur in one epoch?

Accepted Answer

100

Answer

10

Answer

1000

Answer

10000

Question 11

Why is mini-batch gradient descent often preferred on GPUs over pure SGD?

Accepted Answer

Because mini-batches exploit GPU parallel matrix operations

Answer

Because pure SGD uses more memory

Answer

Because GPUs cannot do single-example math

Answer

Because mini-batches need no gradients

Question 12

How does increasing the mini-batch size typically affect the noise in gradient estimates?

Accepted Answer

Larger batches reduce noise in the gradient estimate

Answer

Larger batches increase noise

Answer

Batch size has no effect on noise

Answer

Larger batches eliminate the gradient

Question 13

True or false: Batch gradient descent guarantees finding the global minimum for any loss function. Explain.

Accepted Answer

False, only guaranteed for convex losses

Answer

True, always finds the global minimum

Answer

True, but only for SGD

Answer

False, it never converges

Question 14

In terms of computation per parameter update, which is cheaper: one SGD step or one batch gradient descent step?

Accepted Answer

SGD step is cheaper per update

Answer

Batch step is cheaper per update

Answer

They cost exactly the same per update

Answer

SGD requires no computation

Question 15

Why is it important to shuffle the training data between epochs when using mini-batch or stochastic gradient descent?

Accepted Answer

To prevent biased update patterns and ensure representative batches

Answer

To make training slower

Answer

To reduce the dataset size

Answer

Shuffling is never needed

Question 16

Consider a simple mean-squared-error loss. Write the gradient-update rule that distinguishes SGD from batch GD using one example (xi, yi) for a linear model y = w*x.

Accepted Answer

w := w - lr * 2 * (w*xi - yi) * xi (single example)

Answer

w := w + lr * yi only

Answer

w := w - lr * (average over all N examples) (this is full-batch GD)

Answer

w := w * xi - yi

Question 17

What is a common downside of using a very large batch size (close to full batch) in deep learning?

Accepted Answer

They can generalize worse and lose helpful noise

Answer

They use less memory and generalize better

Answer

They make every update wrong

Answer

They are impossible to compute

Question 18

How does the choice between batch and stochastic training affect how quickly you get feedback on whether the model is learning?

Accepted Answer

SGD/mini-batch gives faster, more frequent feedback

Answer

Batch gives faster feedback per epoch

Answer

Both give identical feedback timing

Answer

Neither gives any feedback

Question 19

In SGD, what is the role of a learning rate schedule that decreases over time?

Accepted Answer

It shrinks steps over time to reduce oscillation and aid convergence

Answer

It speeds up by increasing steps near the end

Answer

It keeps the step size constant

Answer

It disables the gradient

Question 20

Is the gradient computed by mini-batch GD an unbiased estimate of the full-batch gradient? Explain briefly.

Accepted Answer

Yes, it is unbiased if sampled uniformly at random

Answer

No, it is always biased

Answer

Yes, but only for batch size 1

Answer

No, it has the wrong expected value

Question 21

Which training approach typically follows the smoothest, most direct path to the minimum on a convex loss surface?

Accepted Answer

Full-batch gradient descent

Answer

Pure SGD

Answer

Mini-batch of size 1

Answer

None of them converge

Question 22

Why might pure batch gradient descent be impractical for a dataset that does not fit in memory?

Accepted Answer

Because batch GD needs all data at once, which may exceed memory

Answer

Because batch GD uses no memory at all

Answer

Because batch GD only works on tiny datasets by design

Answer

Because mini-batch needs even more memory

Question 23

What does the term 'online learning' refer to, and which gradient method does it resemble?

Accepted Answer

Stochastic gradient descent

Answer

Full-batch gradient descent

Answer

No gradient method

Answer

Closed-form solution

Question 24

How does mini-batch size act as a kind of regularizer in neural network training?

Accepted Answer

Small-batch gradient noise acts as implicit regularization

Answer

Large batches add more regularizing noise

Answer

Batch size never affects generalization

Answer

Noise always causes overfitting

Question 25

In the code below, identify whether it implements batch or stochastic gradient descent.

Accepted Answer

Stochastic gradient descent

Answer

Batch gradient descent

Answer

Mini-batch of size 32

Answer

No gradient descent at all

Question 26

When tuning hyperparameters, why does the optimal learning rate often differ between batch and stochastic training?

Accepted Answer

Noisy SGD often needs a smaller/decaying rate than smooth batch GD

Answer

The learning rate is identical regardless of method

Answer

Batch GD always needs the smallest rate

Answer

Learning rate is unrelated to batch size

Question 27

What happens to the total number of gradient computations per epoch as you change batch size, assuming a fixed dataset?

Accepted Answer

It stays roughly the same (about N per epoch)

Answer

It grows with larger batches

Answer

It shrinks to one per epoch always

Answer

It becomes infinite

Question 28

Why can stochastic gradient descent start making useful progress before it has even seen the entire dataset once?

Accepted Answer

It updates after each example, so it improves immediately

Answer

It must finish an epoch before any update

Answer

It only updates at the end of training

Answer

It never makes progress early

Question 29

A student claims 'batch size 1 always trains a model faster in wall-clock time than batch size 256.' Why is this often wrong on modern hardware?

Accepted Answer

Parallel hardware makes larger batches faster in wall-clock time

Answer

Batch size 1 always wins on any hardware

Answer

Batch size has no effect on speed

Answer

Batch size 256 cannot run on a GPU

Question 30

What is the expected relationship between the gradient noise (standard deviation) and the mini-batch size m, approximately?

Accepted Answer

Noise scales as 1/sqrt(m)

Answer

Noise scales as m

Answer

Noise scales as m squared

Answer

Noise is independent of m

Test yourself