Weight initialization quiz

Question 1

What happens if you initialize all weights in a neural network to exactly zero?

Accepted Answer

All neurons stay identical and the network fails to learn diverse features

Answer

The network trains faster because all gradients are perfectly aligned

Answer

Gradients become very large, causing weight explosion

Answer

The loss instantly reaches zero

Question 2

What does 'symmetry breaking' mean in neural network weight initialization?

Accepted Answer

Using different random values so each neuron learns unique features

Answer

Setting all weights to the same small constant like 0.01

Answer

Setting all biases to zero at the start

Answer

Dividing weights by the square root of the layer depth

Question 3

What is the problem with initializing all neural network weights with values very close to zero, such as 0.0001?

Accepted Answer

Activations shrink layer by layer and gradients vanish, preventing early layers from learning

Answer

Weights explode during the first epoch of training

Answer

The network converges too quickly to a poor local minimum

Answer

Batch normalization is unable to normalize the activations

Question 4

What goes wrong when neural network weights are initialized with very large values?

Accepted Answer

Saturating activations produce near-zero gradients, or gradients explode, destabilizing training

Answer

The network converges faster because it starts exploring a wide region

Answer

Dropout becomes more effective at preventing overfitting

Answer

Batch normalization stops computing meaningful statistics

Question 5

What is the main goal of Xavier (Glorot) initialization?

Accepted Answer

Keep the variance of activations and gradients constant across all layers

Answer

Maximize the initial loss so the network has room to rapidly improve

Answer

Initialize every weight to the same small value to avoid asymmetry

Answer

Match the weight scale exactly to the chosen learning rate

Question 6

In Xavier (Glorot) uniform initialization, weights are drawn from U(−a, +a). What is the formula for a?

Accepted Answer

a = sqrt(6 / (fan_in + fan_out))

Answer

a = sqrt(2 / fan_in)

Answer

a = sqrt(1 / fan_out)

Answer

a = sqrt(fan_in × fan_out)

Question 7

Why is He (Kaiming) initialization preferred over Xavier initialization when the network uses ReLU activations?

Accepted Answer

ReLU zeros ~half of activations, so He doubles the variance to compensate

Answer

He uses a smaller variance making weights safer for gradient descent

Answer

He initialization only works correctly with sigmoid activations

Answer

Xavier was derived later and was found to be inferior for all activations

Question 8

What is the variance used in He (Kaiming) initialization, and for which activation function is it specifically designed?

Accepted Answer

Variance = 2/fan_in, designed for ReLU and its variants

Answer

Variance = 1/fan_in, designed for tanh activations

Answer

Variance = 6/(fan_in + fan_out), designed for sigmoid

Answer

Variance = 1/(fan_in × fan_out), designed for softmax output

Question 9

Which of the following NumPy snippets correctly implements Xavier uniform initialization for a weight matrix?

Accepted Answer

a = np.sqrt(6/(fan_in+fan_out)); W = np.random.uniform(-a, a, (fan_in, fan_out))

Answer

W = np.random.randn(fan_in, fan_out)

Answer

W = np.zeros((fan_in, fan_out))

Answer

W = np.random.uniform(-1, 1, (fan_in, fan_out))

Question 10

What weight initialization does PyTorch's nn.Linear layer use by default?

Accepted Answer

Kaiming (He) uniform initialization

Answer

All-zeros initialization for both weights and biases

Answer

Xavier (Glorot) normal initialization

Answer

Standard normal N(0, 1) with no scaling

Question 11

Why is it safe and standard to initialize biases to zero, even though weights must not all be zero?

Accepted Answer

Biases don't cause the symmetry problem, so zero is a safe and convenient default

Answer

Zero biases reduce backpropagation computation cost

Answer

Non-zero bias initialization always causes exploding gradients

Answer

PyTorch enforces zero bias initialization at the framework level

Question 12

How does poor weight initialization directly contribute to the vanishing gradient problem in a deep network?

Accepted Answer

Small weights cause gradients to shrink multiplicatively through each layer during backprop

Answer

Large weights always cause gradients to vanish through saturation

Answer

Vanishing gradients only arise when batch normalization is applied

Answer

The problem only affects the output layer's gradient

Question 13

What is LeCun initialization, and for which type of network and activation was it originally designed?

Accepted Answer

Variance = 1/fan_in, designed for SELU and self-normalizing networks

Answer

Variance = 2/fan_in, designed for ReLU activations

Answer

Variance = 6/(fan_in + fan_out), designed for tanh

Answer

Variance = fan_in/fan_out, designed for attention layers

Question 14

What is orthogonal initialization, and what key property makes it attractive for very deep networks?

Accepted Answer

The weight matrix satisfies W^T W = I, preserving activation norms through each layer

Answer

All weights are set equal, making every neuron's gradient uniform

Answer

Each neuron's weights are orthogonal to its bias vector

Answer

Weights are sampled uniformly from the surface of a unit hypersphere

Question 15

Define fan_in and fan_out for a fully-connected layer and explain why both appear in the Xavier formula.

Accepted Answer

fan_in = number of input units to the layer; fan_out = number of output units

Answer

fan_in = batch size; fan_out = number of hidden layers

Answer

fan_in = learning rate; fan_out = weight decay coefficient

Answer

fan_in = total model parameters; fan_out = training set size

Question 16

Why does the choice of activation function affect which weight initialization scheme should be used?

Accepted Answer

Each activation scales output variance differently, so initialization must compensate

Answer

Different activations require different learning rates, which in turn changes the init scale

Answer

Activation functions change the effective batch size needed during training

Answer

Different activations require different loss functions, altering initialization requirements

Question 17

Why is initializing the recurrent weight matrix of an RNN to the identity matrix (or near-identity) sometimes beneficial?

Accepted Answer

It copies the hidden state unchanged through time, stabilizing gradient flow early in training

Answer

It sets recurrent weights to zero, preventing interference between time steps

Answer

It makes the RNN mathematically equivalent to a feedforward network

Answer

It maximizes entropy of initial hidden states to encourage exploration

Question 18

How does batch normalization reduce a neural network's sensitivity to the choice of weight initialization?

Accepted Answer

It normalizes activations layer by layer, decoupling training from the initial weight scale

Answer

Batch normalization completely replaces the need for any weight initialization

Answer

It automatically increases the learning rate when initialization is poor

Answer

It keeps all weights positive, reducing the impact of negative initial values

Question 19

How is transfer learning related to the concept of weight initialization?

Accepted Answer

Pre-trained weights from a related task serve as a much better starting initialization

Answer

Transfer learning initializes all weights to zero before fine-tuning begins

Answer

Transfer learning applies Xavier initialization to every layer before training

Answer

Transfer learning eliminates the need for gradient-based optimization

Question 20

Why do practitioners always set a random seed before initializing neural network weights in experiments?

Accepted Answer

They ensure reproducibility so experiments can be fairly compared across runs

Answer

Random seeds speed up weight initialization computation

Answer

They force the model to automatically use Xavier initialization

Answer

They prevent overfitting by randomizing the weight scale at each epoch

Question 21

What is sparse initialization, and in what scenario might it be useful?

Accepted Answer

Most weights start at zero; only a fixed few connections per neuron get random values

Answer

All weights are initialized to very small but non-zero values uniformly

Answer

Weights are sampled from a probability distribution with very heavy tails

Answer

Sparse initialization is equivalent to applying dropout at the initialization step

Question 22

How does the quality of weight initialization affect the speed of training convergence?

Accepted Answer

Good initialization keeps gradients healthy, enabling effective learning from the very first step

Answer

Weight initialization only affects final model accuracy, never training speed

Answer

Poor initialization always leads to faster early convergence due to larger gradient magnitudes

Answer

Initialization only matters for the output layer; hidden layers are insensitive

Question 23

What is the relationship between 'Kaiming initialization' and 'He initialization'?

Accepted Answer

They are the same method — Kaiming is the first name of He, the paper's first author

Answer

Kaiming uses a uniform distribution; He uses a normal distribution

Answer

Kaiming is for CNNs; He initialization is for fully connected networks

Answer

Kaiming suits shallow networks; He suits networks with skip connections

Question 24

Xavier initialization has two variants — Glorot normal and Glorot uniform. How do they differ and does it usually matter?

Accepted Answer

Both have the same variance; normal uses Gaussian sampling, uniform uses a bounded range

Answer

Normal uses variance 1/fan_in; uniform uses variance 1/fan_out

Answer

Glorot normal is for CNNs; Glorot uniform is for recurrent networks

Answer

Glorot uniform always converges faster because it avoids extreme weight values

Question 25

A layer has fan_in = 256 weights connected to each neuron. If each weight has variance σ² and each input has unit variance, what is the variance of the pre-activation sum?

Accepted Answer

Var(pre-activation) = fan_in × σ² = 256σ²

Answer

Var(pre-activation) = σ²

Answer

Var(pre-activation) = σ² / fan_in

Answer

Var(pre-activation) = sqrt(fan_in) × σ²

Question 26

How can large initial weight magnitudes contribute to the 'dying ReLU' problem?

Accepted Answer

Large negative initial weights push pre-activations permanently below zero, killing those ReLU neurons

Answer

Large initial weights force all neurons to output positive values, not negative

Answer

Dying ReLU only occurs with sigmoid activations, not with ReLU

Answer

Large initial weights increase the effective learning rate, causing gradient overshooting

Question 27

For a convolutional layer with a k×k kernel and C_in input channels, what is the fan_in used in He initialization?

Accepted Answer

fan_in = k × k × C_in (kernel area × input channels)

Answer

fan_in = C_in only (number of input channels)

Answer

fan_in = k × k only (the spatial kernel area)

Answer

fan_in = C_in × C_out (input channels × output channels)

Question 28

What does LSUV (Layer-Sequential Unit Variance) initialization do, and why is it considered a data-driven approach?

Accepted Answer

It applies orthogonal init then rescales each layer to unit output variance using real data

Answer

It initializes weights using singular values of the training data covariance matrix

Answer

It is equivalent to applying He initialization one layer at a time

Answer

It uses LSTM gating to adaptively control variance per layer

Question 29

What problem does Fixup initialization solve, and how does it enable training very deep ResNets without batch normalization?

Accepted Answer

It scales residual branch weights by a depth-dependent factor and zeros the last layer of each branch

Answer

Fixup adds extra batch normalization layers at the beginning of each residual block

Answer

Fixup applies He initialization with a higher variance to compensate for skip connections

Answer

It replaces residual connections with identity-initialized dense skip connections

Question 30

In a 10-layer network with tanh activations and weights from N(0, 1), what happens to activations by the 10th layer?

Accepted Answer

Activations saturate near ±1, collapsing variance and causing near-zero gradients

Answer

Variance stays close to 1.0 throughout all 10 layers

Answer

Variance grows exponentially, causing exploding activations by layer 10

Answer

Variance shrinks exactly to zero because tanh has a bounded output range

Test yourself