The math of gradient descent quiz

Question 1

What is the core update rule of gradient descent for a single parameter w with learning rate α?

Accepted Answer

w := w − α·(dL/dw)

Answer

w := w + α·(dL/dw)

Answer

w := w − α·L

Answer

w := α·(dL/dw)

Question 2

Why does gradient descent subtract the gradient rather than add it?

Accepted Answer

Because the gradient points uphill, and we want to go downhill

Answer

Because subtraction is faster to compute

Answer

Because the gradient is always negative

Answer

Because adding would make the learning rate zero

Question 3

For the loss L(w) = w², what is the gradient at w = 3?

Accepted Answer

6

Answer

9

Answer

3

Answer

2

Question 4

What role does the learning rate α play in gradient descent?

Accepted Answer

It controls the size of each step toward the minimum

Answer

It is the number of parameters in the model

Answer

It is the final value of the loss

Answer

It determines the sign of the gradient

Question 5

What can happen if the learning rate is set too large?

Accepted Answer

The loss may oscillate or diverge instead of converging

Answer

Training becomes guaranteed to be faster and more accurate

Answer

The gradient becomes exactly zero

Answer

The model can no longer overfit

Question 6

What does the gradient ∇L of a multivariable loss function represent geometrically?

Accepted Answer

The direction of steepest increase of the loss

Answer

The minimum value of the loss

Answer

The average of all parameters

Answer

The direction of slowest change of the loss

Question 7

For L(w) = w² with learning rate α = 0.1, what is w after one step starting from w = 5?

Accepted Answer

4

Answer

4.5

Answer

6

Answer

0

Question 8

What is the difference between batch gradient descent and stochastic gradient descent (SGD)?

Accepted Answer

Batch uses all data per update; SGD uses one (or few) examples per update

Answer

Batch uses one example; SGD uses all data

Answer

They are identical except in name

Answer

SGD never uses gradients

Question 9

At a minimum of a differentiable loss function, what is the value of the gradient?

Accepted Answer

Zero

Answer

One

Answer

Equal to the learning rate

Answer

Equal to the loss value

Question 10

What is a mini-batch in gradient descent and why is it commonly used?

Accepted Answer

A small subset of data used per update, balancing speed and stability

Answer

The single largest data example

Answer

The final batch of the last epoch only

Answer

A batch containing all data points

Question 11

In gradient descent, what is one epoch?

Accepted Answer

One complete pass through the entire training dataset

Answer

A single parameter update

Answer

The learning rate at step one

Answer

The size of one mini-batch

Question 12

For the loss L(w) = (w − 4)², where is the minimum and what is the gradient there?

Accepted Answer

Minimum at w = 4, gradient 0 there

Answer

Minimum at w = 0, gradient 4 there

Answer

Minimum at w = 2, gradient 1 there

Answer

Minimum at w = 16, gradient 0 there

Question 13

Why is the chain rule essential to gradient descent in neural networks?

Accepted Answer

It lets gradients propagate through composed functions (backpropagation)

Answer

It eliminates the need for a learning rate

Answer

It guarantees a global minimum

Answer

It removes the need for partial derivatives

Question 14

What is a convex function, and why is it ideal for gradient descent?

Accepted Answer

A bowl-shaped function with a single global minimum, ensuring convergence

Answer

A function with many equal local minima

Answer

A function whose gradient is always negative

Answer

A function that has no minimum

Question 15

What problem does the 'vanishing gradient' refer to in deep networks?

Accepted Answer

Gradients shrink toward zero in early layers, stalling learning

Answer

The learning rate grows without bound

Answer

The loss becomes negative

Answer

The dataset disappears during training

Question 16

How does momentum improve plain gradient descent?

Accepted Answer

It uses a running average of past gradients to smooth and speed up updates

Answer

It sets the learning rate to zero

Answer

It removes all noise by using the full dataset

Answer

It ignores the current gradient entirely

Question 17

For two parameters with L(w1, w2) = w1² + w2², what is the gradient vector at (1, 2)?

Accepted Answer

(2, 4)

Answer

(1, 2)

Answer

(4, 2)

Answer

(2, 2)

Question 18

What does it mean for gradient descent to 'converge'?

Accepted Answer

Updates become negligible and the loss stops decreasing

Answer

The learning rate reaches its maximum

Answer

Every parameter equals zero

Answer

The dataset is fully shuffled

Question 19

In mean squared error L = (1/n)Σ(yi − ŷi)², why is the gradient with respect to predictions proportional to the error (ŷi − yi)?

Accepted Answer

Because differentiating the squared error yields a term linear in (ŷi − yi)

Answer

Because MSE has no derivative

Answer

Because errors are always constant

Answer

Because the learning rate equals the error

Question 20

What is the purpose of feature scaling (normalization) before running gradient descent?

Accepted Answer

It reshapes the loss surface so descent converges faster and more directly

Answer

It increases the number of local minima

Answer

It replaces the need for a gradient

Answer

It guarantees zero loss

Question 21

What is a learning rate schedule (decay)?

Accepted Answer

Gradually decreasing the learning rate as training progresses

Answer

Increasing the learning rate to infinity

Answer

Keeping the gradient constant

Answer

Choosing the batch size each epoch

Question 22

Why can a learning rate that is too small be a problem?

Accepted Answer

Training becomes extremely slow and may make negligible progress

Answer

The loss is guaranteed to diverge

Answer

The gradient becomes infinite

Answer

The model cannot compute predictions

Question 23

Write Python that performs 3 gradient descent steps on L(w)=w² starting at w=10 with α=0.1. What is the final w?

Accepted Answer

5.12

Answer

8.0

Answer

6.4

Answer

2.0

Question 24

What is a saddle point, and why can it slow gradient descent?

Accepted Answer

A point with zero gradient that is neither a min nor a max, where descent stalls

Answer

The global minimum of any function

Answer

A point where the learning rate is undefined

Answer

A point where loss is exactly zero

Question 25

How does the Adam optimizer build on basic gradient descent?

Accepted Answer

It combines momentum with per-parameter adaptive learning rates

Answer

It removes gradients entirely

Answer

It uses only the full batch every step

Answer

It fixes the learning rate at exactly 1

Question 26

What does the magnitude (norm) of the gradient tell you during training?

Accepted Answer

How steep the loss surface is at the current point

Answer

The total number of training examples

Answer

The final accuracy of the model

Answer

The learning rate schedule

Question 27

What is gradient clipping and when is it used?

Accepted Answer

Capping gradient magnitude to prevent exploding, destabilizing updates

Answer

Setting all gradients to zero

Answer

Increasing gradients to speed up training

Answer

Removing the learning rate

Question 28

For L(w) = w² + 4w + 4, at which value of w does gradient descent converge?

Accepted Answer

w = −2

Answer

w = 2

Answer

w = 0

Answer

w = 4

Question 29

Why is the loss function required to be (mostly) differentiable for gradient descent?

Accepted Answer

Because the algorithm needs a gradient (derivative) to choose its direction

Answer

Because differentiable functions are always convex

Answer

Because it makes the learning rate unnecessary

Answer

Because non-differentiable functions have no minimum

Question 30

What is the noisy advantage of stochastic gradient descent compared to full-batch descent?

Accepted Answer

Its noisy updates can help escape shallow local minima and saddle points

Answer

It always finds the exact gradient

Answer

It never requires a learning rate

Answer

It guarantees the global minimum every time

Test yourself