Gradient descent variants (SGD, Adam) quiz

Question 1

What is the core idea behind gradient descent as an optimization algorithm?

Accepted Answer

It moves parameters opposite to the gradient to reduce loss

Answer

It moves parameters along the gradient to increase loss

Answer

It randomly searches the parameter space until loss is zero

Answer

It solves the loss function exactly using linear algebra

Question 2

How does Batch Gradient Descent differ from Stochastic Gradient Descent (SGD) in how it uses training data?

Accepted Answer

Batch uses the whole dataset per update; SGD uses one example per update

Answer

Batch uses one example per update; SGD uses the whole dataset

Answer

Both use the whole dataset but SGD is faster

Answer

Both use one example but batch repeats it many times

Question 3

What is a mini-batch in mini-batch gradient descent, and why is it commonly used?

Accepted Answer

A small subset of data used per update, balancing speed and stability

Answer

The entire dataset used once per epoch

Answer

A single example chosen at random

Answer

A copy of the validation set used for testing

Question 4

What role does the learning rate play in gradient descent?

Accepted Answer

It controls the size of each parameter update step

Answer

It controls how many epochs the model trains for

Answer

It sets the number of layers in the network

Answer

It determines the batch size automatically

Question 5

Why is plain SGD's update path often described as 'noisy' compared to batch gradient descent?

Accepted Answer

The gradient is estimated from few samples, so it varies around the true gradient

Answer

The learning rate randomly changes every step

Answer

The loss function itself is random

Answer

The weights are initialized to random values each step

Question 6

What problem does momentum solve in gradient descent, and how does it work intuitively?

Accepted Answer

It accumulates past gradients to speed up and dampen oscillations

Answer

It increases the learning rate every epoch

Answer

It reduces the number of parameters

Answer

It randomly resets the gradient to avoid local minima

Question 7

In the momentum update v = β·v + (1-β)·g, what does the hyperparameter β control?

Accepted Answer

How much past gradients are remembered versus the current gradient

Answer

The total number of training steps

Answer

The size of the mini-batch

Answer

The dimension of the parameter vector

Question 8

What is the key idea behind adaptive learning rate methods like AdaGrad?

Accepted Answer

Each parameter gets a per-parameter learning rate based on its gradient history

Answer

All parameters share a single fixed learning rate

Answer

The learning rate increases for large gradients

Answer

It removes the need for a loss function

Question 9

What weakness of AdaGrad does RMSProp fix, and how?

Accepted Answer

It uses a decaying average of squared gradients so the rate does not vanish

Answer

It removes the learning rate entirely

Answer

It accumulates squared gradients faster

Answer

It uses only the most recent single gradient

Question 10

Adam combines ideas from two other methods. Which two, and what does each contribute?

Accepted Answer

Momentum (first moment) and RMSProp (second moment)

Answer

AdaGrad and Newton's method

Answer

Batch GD and dropout

Answer

L1 and L2 regularization

Question 11

In Adam, what do the first moment (m) and second moment (v) estimates represent?

Accepted Answer

m is the average gradient; v is the average squared gradient

Answer

m is the loss; v is the accuracy

Answer

m is the learning rate; v is the batch size

Answer

m is the weight; v is the bias

Question 12

Why does Adam apply 'bias correction' to its moment estimates?

Accepted Answer

Because zero-initialized moments are biased low early in training

Answer

Because the gradients are always biased high

Answer

To make the learning rate increase over time

Answer

To remove the need for a learning rate

Question 13

What are the typical default values of Adam's hyperparameters β1, β2, and ε?

Accepted Answer

β1=0.9, β2=0.999, ε=1e-8

Answer

β1=0.5, β2=0.5, ε=1.0

Answer

β1=0.999, β2=0.9, ε=0.1

Answer

β1=0.1, β2=0.2, ε=10

Question 14

What is the purpose of the small constant ε (epsilon) in the Adam and RMSProp update rules?

Accepted Answer

It prevents division by zero for numerical stability

Answer

It controls the momentum strength

Answer

It sets the number of epochs

Answer

It scales the loss function

Question 15

Why is Adam often preferred over plain SGD as a default optimizer for deep neural networks?

Accepted Answer

It converges fast and needs less learning-rate tuning

Answer

It always reaches a lower loss than any other method

Answer

It uses no hyperparameters at all

Answer

It requires less memory than SGD

Question 16

What can happen if the learning rate in gradient descent is set far too high?

Accepted Answer

The loss may oscillate or diverge instead of decreasing

Answer

Training becomes extremely slow but stable

Answer

The model trains perfectly with no downside

Answer

The gradient becomes exactly zero

Question 17

What is a learning rate schedule, and why might you use one?

Accepted Answer

A plan to change the learning rate during training, often decreasing it

Answer

A fixed number of epochs to train

Answer

A way to schedule which GPU is used

Answer

A method to pick the batch size

Question 18

In one SGD update step, write the parameter update rule using learning rate η and gradient g.

Accepted Answer

θ = θ - η·g

Answer

θ = θ + η·g

Answer

θ = η·g - θ

Answer

θ = g - η·θ

Question 19

How does the choice of mini-batch size affect training, in terms of gradient noise and hardware use?

Accepted Answer

Larger batches reduce noise but cost more memory; smaller batches add helpful noise

Answer

Batch size has no effect on gradient noise

Answer

Smaller batches always converge faster and better

Answer

Larger batches always generalize better

Question 20

What is an 'epoch' in the context of gradient-descent-based training?

Accepted Answer

One full pass through the entire training dataset

Answer

A single parameter update

Answer

The size of one mini-batch

Answer

The learning rate at the start of training

Question 21

What does it mean for an optimizer to get 'stuck' at a saddle point, and why is it a concern in deep learning?

Accepted Answer

A zero-gradient point that is neither a min nor max, slowing progress

Answer

A point where the loss is exactly zero

Answer

The global minimum of the loss

Answer

A point where the learning rate becomes negative

Question 22

How does momentum help an optimizer move through flat regions or small bumps in the loss surface?

Accepted Answer

Accumulated velocity carries it across flat or bumpy regions

Answer

It increases the gradient magnitude artificially

Answer

It deletes flat regions from the loss surface

Answer

It switches to batch gradient descent automatically

Question 23

What is the difference between classical momentum and Nesterov Accelerated Gradient (NAG)?

Accepted Answer

Nesterov computes the gradient after a lookahead step in the velocity direction

Answer

Nesterov ignores the velocity term entirely

Answer

Classical momentum uses adaptive learning rates

Answer

They are identical algorithms with different names

Question 24

In PyTorch, how do you create an Adam optimizer for a model and perform one optimization step?

Accepted Answer

Create optim.Adam(params, lr), then zero_grad, backward, step

Answer

Just call model.train() repeatedly

Answer

Manually subtract gradients with a for loop only

Answer

Use optim.Adam but never call step()

Question 25

Why must you call optimizer.zero_grad() (or equivalent) before backpropagation in each training step?

Accepted Answer

Gradients accumulate by default, so they must be reset each step

Answer

It frees GPU memory permanently

Answer

It resets the model weights to zero

Answer

It changes the learning rate

Question 26

What is gradient clipping, and which training problem does it address?

Accepted Answer

Capping gradient magnitude to prevent exploding gradients

Answer

Removing small gradients to speed up training

Answer

Setting all gradients to the same value

Answer

Clipping the learning rate to a maximum

Question 27

Why does SGD sometimes generalize better than Adam despite Adam often training faster?

Accepted Answer

SGD tends to find flatter minima that generalize better

Answer

Adam ignores the training data

Answer

SGD always trains faster than Adam

Answer

Adam cannot reach a low training loss

Question 28

What problem with Adam does AdamW specifically fix regarding weight decay?

Accepted Answer

It decouples weight decay from the adaptive gradient update

Answer

It removes momentum from Adam

Answer

It makes Adam use the full batch

Answer

It disables the second moment estimate

Question 29

If your training loss decreases steadily but then suddenly spikes to NaN, what is a likely optimizer-related cause?

Accepted Answer

A learning rate too high causing exploding/unstable updates

Answer

The batch size is too small

Answer

Too many epochs were used

Answer

The model has too few parameters

Question 30

Why is a 'warmup' phase sometimes used at the start of training with optimizers like Adam?

Accepted Answer

To start with small steps while early gradient estimates are unreliable

Answer

To skip training on the first epoch

Answer

To increase the batch size gradually

Answer

To freeze the model weights initially

Test yourself