Gradient descent and optimization quiz

Question 1

What is gradient descent, in one sentence?

Accepted Answer

An iterative method that minimizes a function by moving opposite to the gradient

Answer

An algorithm that maximizes a function by following the gradient

Answer

A method that sorts data by gradient magnitude

Answer

A technique for randomly searching parameter space

Question 2

In the update rule θ = θ − α·∇J(θ), what does α represent and why must it be chosen carefully?

Accepted Answer

The learning rate; too large diverges, too small converges slowly

Answer

The number of training samples; it must match the dataset size

Answer

The gradient itself; it must always be positive

Answer

The momentum coefficient; it must equal 0.9

Question 3

Why do we move in the direction OPPOSITE to the gradient when minimizing a function?

Accepted Answer

Because the gradient points toward steepest increase, so its opposite decreases the function

Answer

Because the gradient points toward the minimum directly

Answer

Because gradients are always negative

Answer

Because it makes the math simpler but has no real meaning

Question 4

What is the difference between batch gradient descent and stochastic gradient descent (SGD)?

Accepted Answer

Batch uses the full dataset per update; SGD uses one sample per update

Answer

Batch uses one sample per step; SGD uses all samples

Answer

They are identical except for the learning rate

Answer

SGD never uses gradients, only random guesses

Question 5

What is mini-batch gradient descent and why is it the most common choice in deep learning?

Accepted Answer

It uses small subsets, balancing stability, speed, and GPU efficiency

Answer

It uses the whole dataset and is slowest

Answer

It uses only the first sample of the dataset

Answer

It avoids gradients to save memory

Question 6

What is a local minimum, and why can it be a problem for gradient descent?

Accepted Answer

A point lower than nearby points but maybe not the global best, where descent can get stuck

Answer

A point higher than all neighbors where descent speeds up

Answer

The starting point of optimization

Answer

A point where the learning rate becomes zero

Question 7

What is a convex function, and why is gradient descent guaranteed to find the global minimum on one?

Accepted Answer

A bowl-shaped function with one global minimum and no other local minima

Answer

A function with many local minima where descent fails

Answer

A function that is always increasing

Answer

Any function whose gradient is constant

Question 8

What does the learning rate being TOO HIGH typically cause during training?

Accepted Answer

Overshooting the minimum, causing loss to oscillate or diverge

Answer

Extremely slow but smooth convergence

Answer

No effect on training at all

Answer

Guaranteed convergence to the global minimum

Question 9

What is momentum in gradient descent, and what problem does it help solve?

Accepted Answer

It adds a fraction of the previous update, accelerating descent and damping oscillations

Answer

It randomly resets parameters to escape minima

Answer

It reduces the dataset size each step

Answer

It forces the gradient to be zero

Question 10

Briefly, what makes the Adam optimizer popular compared to plain SGD?

Accepted Answer

It combines momentum with adaptive per-parameter learning rates for fast, robust convergence

Answer

It ignores gradients entirely

Answer

It only works on convex functions

Answer

It requires the full dataset for every update

Question 11

What is an epoch in the context of training with gradient descent?

Accepted Answer

One complete pass through the entire training dataset

Answer

A single parameter update

Answer

The final converged model

Answer

The size of one mini-batch

Question 12

For the function f(x) = x², the derivative is f'(x) = 2x. Starting at x = 4 with learning rate α = 0.1, what is x after one gradient descent step?

Accepted Answer

3.2

Answer

4.8

Answer

0.8

Answer

3.6

Question 13

What does it mean for gradient descent to 'converge'?

Accepted Answer

Updates become tiny and loss stops decreasing meaningfully near a minimum

Answer

The loss starts increasing rapidly

Answer

The dataset is fully memorized

Answer

The learning rate reaches one

Question 14

Why is feature scaling (normalization) often important before running gradient descent?

Accepted Answer

It makes the loss surface more symmetric so descent converges faster

Answer

It changes the global minimum location

Answer

It is only needed for classification, never regression

Answer

It eliminates the need for a learning rate

Question 15

What is a saddle point, and why is it relevant in high-dimensional optimization?

Accepted Answer

A point with zero gradient that is a min in some directions and max in others

Answer

A point that is the global minimum

Answer

A point where the learning rate is highest

Answer

Any point where the loss is exactly zero

Question 16

What is the vanishing gradient problem in training deep networks?

Accepted Answer

Gradients shrink toward zero through layers, so early layers barely learn

Answer

Gradients grow too large and explode

Answer

The loss function disappears entirely

Answer

The learning rate becomes negative

Question 17

What is a learning rate schedule (decay), and why is it useful?

Accepted Answer

Gradually reducing the learning rate so early steps are big and later steps fine-tune

Answer

Increasing the learning rate over time to speed up

Answer

Keeping the learning rate fixed forever

Answer

Setting the learning rate to zero immediately

Question 18

What is the cost (loss) function's role in gradient descent?

Accepted Answer

It measures prediction error and provides the gradient that guides updates

Answer

It stores the training data

Answer

It sets the learning rate automatically

Answer

It counts the number of parameters

Question 19

In SGD, why are the updates 'noisy', and can this noise be beneficial?

Accepted Answer

Single-sample gradient estimates vary, and the noise can help escape shallow minima

Answer

Noise comes from rounding errors and is always harmful

Answer

SGD adds random numbers on purpose for no reason

Answer

There is no noise in SGD

Question 20

What is the exploding gradient problem, and one common way to address it?

Accepted Answer

Gradients grow huge and destabilize training; fixed by gradient clipping

Answer

Gradients become zero; fixed by increasing learning rate

Answer

The dataset becomes too large; fixed by sampling

Answer

The model has too few parameters; fixed by adding layers

Question 21

Why is the derivative/gradient being zero a necessary condition at a minimum?

Accepted Answer

Because the slope is flat (zero) at a minimum, so updates stop

Answer

Because the function value is always zero at a minimum

Answer

Because the learning rate must be zero

Answer

Because the loss is negative there

Question 22

What is the difference between a global minimum and a local minimum?

Accepted Answer

Global is the lowest over the whole domain; local is lowest only nearby

Answer

They are always the same point

Answer

Local is always lower than global

Answer

Global minimum only exists for non-convex functions

Question 23

In the Python loop below, what role does `grad` play and what is the gradient of the loss L = (w-3)²?

Accepted Answer

grad is the derivative 2*(w-3) guiding updates toward w=3

Answer

grad is the loss value; the gradient is (w-3)

Answer

grad is the learning rate; the gradient is constant

Answer

grad is random noise

Question 24

What is early stopping, and how does it relate to optimization and overfitting?

Accepted Answer

Stopping when validation performance stops improving, to prevent overfitting

Answer

Stopping when the GPU runs out of memory

Answer

Stopping after exactly one epoch always

Answer

Stopping when the learning rate reaches zero

Question 25

Why can gradient descent be slow in long, narrow 'ravine' shaped loss surfaces?

Accepted Answer

Because gradients point across the narrow direction, causing slow zigzagging

Answer

Because the gradient is always zero in a ravine

Answer

Because ravines have no minimum

Answer

Because the learning rate automatically drops to zero

Question 26

What does 'batch size' control, and what is one trade-off of using a very large batch size?

Accepted Answer

It sets samples per update; large batches use more memory and may generalize worse

Answer

It controls the learning rate; large means faster always

Answer

It sets the number of epochs

Answer

It has no effect on training

Question 27

How does the chain rule from calculus underpin backpropagation in neural networks?

Accepted Answer

It uses the chain rule to multiply derivatives layer by layer back through the network

Answer

It randomly assigns gradients to weights

Answer

It only works for a single-layer network

Answer

It avoids derivatives entirely

Question 28

What is the purpose of the gradient's MAGNITUDE (not just its direction) in an update step?

Accepted Answer

Larger magnitude means steeper slope and bigger steps, shrinking near minima

Answer

Magnitude is ignored; only direction matters

Answer

Magnitude sets the number of epochs

Answer

Magnitude always equals the learning rate

Question 29

What is RMSProp, and what problem with AdaGrad does it address?

Accepted Answer

It uses a decaying average of squared gradients to stop the learning rate from shrinking to zero

Answer

It removes the learning rate entirely

Answer

It only works on convex problems

Answer

It increases the gradient magnitude over time

Question 30

If the loss curve goes down then suddenly spikes upward during training, what is a likely cause?

Accepted Answer

An overly high learning rate or exploding gradients causing an unstable step

Answer

The dataset is too small

Answer

The model has converged perfectly

Answer

Too many epochs were used

Test yourself