Gradient vanishing and exploding quiz

Question 1

What is the 'vanishing gradient problem' in deep neural networks?

Accepted Answer

Gradients shrink toward zero in early layers, stalling their learning

Answer

Gradients grow uncontrollably and cause overflow

Answer

The model has too many neurons to fit in memory

Answer

The dataset is too small to train on

Question 2

What is the 'exploding gradient problem'?

Accepted Answer

Gradients grow very large, causing unstable updates and possible NaN

Answer

Gradients become very small and vanish

Answer

The network stops using activation functions

Answer

The learning rate automatically drops to zero

Question 3

Why are deep networks more prone to vanishing and exploding gradients than shallow ones?

Accepted Answer

The chain rule multiplies many factors, so products shrink or grow exponentially with depth

Answer

Deeper networks always overfit

Answer

Deeper networks use larger learning rates by default

Answer

More layers means fewer parameters to update

Question 4

How does the chain rule of calculus explain vanishing and exploding gradients?

Accepted Answer

The chain rule multiplies many derivatives, so the product can shrink or grow exponentially

Answer

The chain rule adds derivatives, so errors cancel out

Answer

The chain rule only applies to the output layer

Answer

The chain rule prevents gradients from changing

Question 5

What is the maximum value of the derivative of the sigmoid activation function, and why does it matter?

Accepted Answer

0.25

Answer

1.0

Answer

0.5

Answer

0.1

Question 6

Why does the ReLU activation help reduce the vanishing gradient problem?

Accepted Answer

Its derivative is 1 for positive inputs, so gradients are not shrunk there

Answer

It squashes all values between 0 and 1

Answer

It multiplies gradients by 0.25

Answer

It removes the need for backpropagation

Question 7

What is the 'dead ReLU' problem and how is it related to gradient flow?

Accepted Answer

A neuron always outputs zero and has zero gradient, so it never updates

Answer

A neuron outputs only very large numbers

Answer

A neuron that has too many connections

Answer

A neuron whose learning rate is negative

Question 8

How does Leaky ReLU address the dead ReLU / vanishing gradient issue?

Accepted Answer

It gives a small nonzero slope for negative inputs, keeping gradients alive

Answer

It clips all outputs to 1

Answer

It removes the negative inputs entirely

Answer

It doubles the gradient on positive inputs

Question 9

What is gradient clipping and which problem does it primarily solve?

Accepted Answer

It caps gradient magnitude to prevent exploding gradients

Answer

It increases gradients to fix vanishing

Answer

It removes all negative gradients

Answer

It deletes layers with small gradients

Question 10

How do residual (skip) connections in networks like ResNet help with vanishing gradients?

Accepted Answer

They add an identity shortcut so gradients can flow without shrinking

Answer

They remove activation functions entirely

Answer

They reduce the number of layers

Answer

They clip the gradient to a fixed value

Question 11

How does batch normalization help mitigate vanishing and exploding gradients?

Accepted Answer

It normalizes layer inputs, keeping activations and gradients in a stable range

Answer

It deletes saturated neurons

Answer

It increases the learning rate automatically

Answer

It replaces backpropagation with a faster method

Question 12

What is the purpose of Xavier (Glorot) weight initialization?

Accepted Answer

To scale initial weight variance so signal variance stays stable across layers

Answer

To set all weights to zero

Answer

To make all weights identical

Answer

To remove the bias terms

Question 13

Why is He initialization preferred over Xavier initialization for ReLU networks?

Accepted Answer

It uses variance 2/n_in to compensate for ReLU zeroing half the inputs

Answer

It sets weights to exactly 2

Answer

It only works without activation functions

Answer

It always produces smaller weights than Xavier

Question 14

Why do recurrent neural networks (RNNs) especially suffer from vanishing and exploding gradients?

Accepted Answer

Repeated multiplication of the same recurrent weights over time shrinks or grows gradients

Answer

They have no activation functions

Answer

They use too few time steps

Answer

They cannot use backpropagation

Question 15

How do LSTM networks help combat the vanishing gradient problem in sequences?

Accepted Answer

By using a gated cell state that lets gradients flow nearly unchanged over time

Answer

By removing recurrence completely

Answer

By clipping every weight to 1

Answer

By using only sigmoid activations

Question 16

What is a common symptom that exploding gradients are occurring during training?

Accepted Answer

The loss becomes NaN or oscillates wildly with huge weight values

Answer

The loss stays perfectly flat at zero

Answer

The model trains slightly faster than expected

Answer

Accuracy reaches 100% immediately

Question 17

What is a common symptom that vanishing gradients are occurring during training?

Accepted Answer

Early-layer weights barely change and the loss plateaus

Answer

The loss instantly becomes NaN

Answer

Weights grow to enormous values

Answer

Training finishes in a single step

Question 18

How does the choice of learning rate interact with the exploding gradient problem?

Accepted Answer

A high learning rate magnifies gradients into unstable, diverging updates

Answer

Learning rate has no effect on gradients

Answer

A high learning rate always fixes exploding gradients

Answer

Learning rate only matters for the output layer

Question 19

What does it mean for an activation function to be 'saturated', and how does it relate to vanishing gradients?

Accepted Answer

The activation output is flat and its derivative is near zero, blocking gradient flow

Answer

The activation outputs random noise

Answer

The activation grows without bound

Answer

The activation ignores its input weights

Question 20

Compared with sigmoid, why might tanh be slightly better for gradient flow, and why is it still imperfect?

Accepted Answer

Tanh is zero-centered with max derivative 1, but still saturates for large inputs

Answer

Tanh never saturates

Answer

Tanh has a derivative always greater than 1

Answer

Tanh has no derivative

Question 21

In a simple deep linear network, if each layer multiplies the gradient by a factor of 0.5, what happens to the gradient after 10 layers?

Accepted Answer

It is multiplied by about 0.001 (0.5^10)

Answer

It stays the same

Answer

It doubles

Answer

It becomes 10 times larger

Question 22

In a simple deep network, if each layer multiplies the gradient by a factor of 2, what happens after 10 layers?

Accepted Answer

It is multiplied by 1024 (2^10)

Answer

It halves

Answer

It stays constant

Answer

It becomes zero

Question 23

How does monitoring the gradient norm per layer help diagnose vanishing or exploding gradients?

Accepted Answer

Tiny norms in early layers signal vanishing; huge norms signal exploding

Answer

It tells you the model's accuracy directly

Answer

It measures only the output prediction

Answer

It has no relation to gradient problems

Question 24

What is truncated backpropagation through time (TBPTT) and how does it relate to gradient problems in RNNs?

Accepted Answer

It propagates gradients over a limited number of time steps to reduce instability

Answer

It removes the recurrent connections

Answer

It doubles the sequence length

Answer

It only works for feedforward networks

Question 25

Why does poor weight initialization with very large values tend to cause exploding gradients?

Accepted Answer

Large weights make per-layer factors exceed one, so gradients grow exponentially

Answer

Large weights guarantee faster convergence

Answer

Large weights make all gradients zero

Answer

Initialization does not affect gradients

Question 26

How do gated architectures like the GRU help with gradient flow compared to vanilla RNNs?

Accepted Answer

By using gates that maintain a near-identity path for gradients across time

Answer

By removing the hidden state

Answer

By forcing all gradients to 1

Answer

By using larger learning rates

Question 27

Which statement best describes WHERE vanishing gradients hurt most in a deep feedforward network?

Accepted Answer

The earliest layers, because their gradients pass through the most factors

Answer

Only the output layer

Answer

Only the bias terms

Answer

No specific layer is affected more

Question 28

How can the spectral properties (eigenvalues) of the recurrent weight matrix predict vanishing or exploding gradients in an RNN?

Accepted Answer

Spectral radius below 1 favors vanishing; above 1 favors exploding gradients

Answer

Eigenvalues have no effect on gradients

Answer

Only negative eigenvalues cause exploding

Answer

Spectral radius determines the learning rate

Question 29

Why do ELU or SELU activations help maintain healthy gradient flow?

Accepted Answer

They keep activations near zero-mean and avoid blocking gradients, aiding flow

Answer

They set all negative inputs to large positive numbers

Answer

They eliminate the need for weights

Answer

They always cause exploding gradients

Question 30

How does layer normalization differ from batch normalization in stabilizing gradients, especially for RNNs?

Accepted Answer

It normalizes across features of one example, working well for RNNs and small batches

Answer

It only works with very large batches

Answer

It removes normalization entirely

Answer

It is identical to batch normalization

Test yourself