Weight initialization

Every neuron in a layer computes the same output and receives identical gradients during backpropagation, so all neurons update identically throughout training. This is called the symmetry problem: the network never learns diverse features, making extra neurons completely redundant. Weight initialization must break this symmetry by assigning different values to different weights.

Symmetry breaking means assigning different random values to different weights so that neurons in the same layer compute different outputs and learn different features from the start. Without symmetry breaking, every neuron in a layer is mathematically identical throughout training, wasting model capacity. Random initialization is the standard way to achieve this.

Very small weights cause activations to shrink layer by layer as signals pass through the network forward. During backpropagation, gradients are multiplied by these tiny weights repeatedly, causing them to become negligibly small in early layers — this is the vanishing gradient problem. Early layers then barely update, preventing the deep network from learning.

With sigmoid or tanh activations, large pre-activation values push the neuron into the saturation region where the derivative is near zero, producing vanishing gradients despite large weights. In networks without saturating activations, large weights can cause exploding gradients during backpropagation, making training diverge. Both failure modes make learning unstable or impossible.

Xavier initialization aims to keep the variance of activations and gradients roughly constant across all layers, preventing signals from shrinking or growing as they travel forward and backward through the network. It achieves this by scaling weights according to the number of inputs and outputs of each layer, using variance = 2/(fan_in + fan_out). This balances the forward and backward flow of information.

The value of a is sqrt(6 / (fan_in + fan_out)), where fan_in is the number of input neurons and fan_out is the number of output neurons of the layer. This choice gives the weight matrix a variance of 2/(fan_in + fan_out), which is derived by requiring that the variance of activations is preserved on average through both the forward and backward passes.

import numpy as np
fan_in, fan_out = 128, 64
a = np.sqrt(6 / (fan_in + fan_out))  # = sqrt(6/192) ≈ 0.177
W = np.random.uniform(-a, a, size=(fan_in, fan_out))

ReLU sets all negative pre-activation values to zero, which effectively zeroes out about half the neurons in each layer and roughly halves the variance of the layer's output. Xavier initialization was derived assuming the activation function preserves variance (appropriate for tanh or sigmoid), so it under-scales weights for ReLU. He initialization compensates by using variance = 2/fan_in, doubling Xavier's scaling to maintain stable variance through deep ReLU networks.

He initialization uses variance = 2/fan_in, where fan_in is the number of input connections to a neuron. It is designed for ReLU and its variants (Leaky ReLU, ELU), because ReLU zeroes approximately half the pre-activation values, reducing output variance by a factor of 2. The factor of 2 in He initialization counteracts this halving, keeping activation variance stable from layer to layer.

import numpy as np
fan_in = 256
std = np.sqrt(2 / fan_in)  # He init standard deviation
W = np.random.randn(fan_in, 128) * std

The correct approach samples from the uniform distribution U(-a, a) where a = sqrt(6 / (fan_in + fan_out)). This ensures the weight variance is 2/(fan_in + fan_out), balancing signal variance in both the forward and backward passes. Using np.random.randn without scaling or using the wrong formula gives incorrect variance.

import numpy as np
fan_in, fan_out = 128, 64

# Correct Xavier uniform:
a = np.sqrt(6 / (fan_in + fan_out))
W = np.random.uniform(-a, a, size=(fan_in, fan_out))
print('Variance:', W.var())  # ≈ 2/(fan_in+fan_out)

PyTorch's nn.Linear layer uses Kaiming (He) uniform initialization for the weight matrix and initializes biases uniformly in [-1/sqrt(fan_in), +1/sqrt(fan_in)]. Internally it calls kaiming_uniform_(a=sqrt(5)), which targets a leaky-ReLU variant, but the result is a sensible general-purpose starting point that works well across most architectures without any manual initialization code.

import torch.nn as nn
layer = nn.Linear(128, 64)
# PyTorch applies kaiming_uniform_(a=sqrt(5)) by default
# Weights ~ U(-1/sqrt(fan_in), +1/sqrt(fan_in)) ≈ U(-0.088, 0.088)
print(layer.weight.shape)  # torch.Size([64, 128])

Biases do not participate in the symmetry problem because each neuron's gradient with respect to its bias depends on that neuron's activation, which differs across neurons once weights are randomly initialized. Initializing biases to zero keeps activations centered initially without causing all neurons to be identical. Weights must be different from each other, but biases can all start at zero without any issue.

During backpropagation, the gradient at each layer is multiplied by the weight matrix (via the chain rule). If weights are initialized too small, these repeated multiplications cause gradients to shrink exponentially as they travel backward through layers. By the time gradients reach the first few layers, they are so close to zero that those layers effectively stop learning.

LeCun initialization uses variance = 1/fan_in, derived so that the variance of activations stays constant when using sigmoid-like or SELU (Scaled Exponential Linear Unit) activations. It is the basis for self-normalizing neural networks: with SELU activations and LeCun normal initialization, deep networks maintain stable mean and variance without batch normalization.

Orthogonal initialization sets the weight matrix W to a random orthogonal matrix — one satisfying W^T W = I (identity). Multiplying a vector by an orthogonal matrix preserves its Euclidean length (norm), so activations neither shrink nor grow as they propagate through the layers. This prevents both vanishing and exploding activations in very deep networks even before any training begins.

fan_in is the number of input connections to each neuron (the input feature dimension), and fan_out is the number of neurons in the layer (output dimension). Xavier initialization uses both because it must simultaneously keep the variance stable in the forward pass (which depends on fan_in) and in the backward pass (which depends on fan_out). Averaging them as 2/(fan_in + fan_out) is a practical compromise.

Different activation functions change the variance of their outputs by different amounts: ReLU halves variance by zeroing negative values, while tanh and sigmoid compress large inputs toward ±1 (saturation). A weight initialization formula must compensate for this scaling so that the overall activation variance stays constant across layers. This is why He (×2) is used for ReLU, Xavier (×1) for tanh, and LeCun (×1) for SELU.

At the start of training, an identity recurrent matrix means the hidden state is passed from one time step to the next without any distortion — it is simply copied. This prevents gradients from vanishing or exploding as they flow backward through many time steps, providing a stable starting point before the RNN learns actual temporal patterns. It is analogous to starting with no temporal memory bias.

Batch normalization re-normalizes the activations after each layer to have approximately zero mean and unit variance, regardless of what the weights are doing. This decouples the signal scale from the weight magnitudes, so even if initialization is poor, activations are brought back to a reasonable range before reaching the next layer. As a result, training proceeds without requiring near-perfect initialization.

In transfer learning, instead of starting from random weights, a model is initialized with weights already pre-trained on a large related dataset (e.g., ImageNet for vision tasks). These pre-trained weights represent a much better starting point in parameter space than random initialization, leading to faster convergence and often higher accuracy, especially when the target dataset is small.

Weight initialization involves random number generation, so different runs produce different starting weights and potentially different results. Setting a fixed random seed ensures the exact same random numbers are generated every time the code runs, making experiments reproducible. This is essential for debugging and for fairly comparing two methods — any result difference must come from the code change, not random chance.

import torch
torch.manual_seed(42)
import numpy as np
np.random.seed(42)
# Now weight init is reproducible every run

Sparse initialization sets most weights to exactly zero and assigns small random non-zero values to only a fixed, small number of connections per neuron. This can be useful in very wide networks to reduce redundancy early in training, as it encourages neurons to specialize by focusing on a limited set of inputs. It can also reduce initial computation when combined with sparse linear algebra operations.

A good initialization places the weights in a region of parameter space where loss gradients are neither vanishing nor exploding, so every gradient step moves the weights meaningfully toward a solution. With poor initialization, the optimizer may spend many steps simply correcting the weight scale before actual task learning begins, or training may diverge entirely. Thus, initialization directly controls how quickly and reliably a network reaches a good solution.

They are exactly the same method. 'Kaiming' is the first name of Kaiming He, the lead author of the 2015 paper 'Delving Deep into Rectifiers' that introduced the technique. Both names are used interchangeably in literature, and deep learning frameworks like PyTorch use the name 'kaiming_uniform_' and 'kaiming_normal_' in their API.

Glorot normal samples weights from a Gaussian distribution N(0, sqrt(2/(fan_in+fan_out))), while Glorot uniform samples from U(-sqrt(6/(fan_in+fan_out)), +sqrt(6/(fan_in+fan_out))). Both have the same variance of 2/(fan_in+fan_out), and in practice the training outcome difference is negligible for most architectures. The choice is mainly a matter of convention or personal preference.

The pre-activation is the sum of fan_in products w_i × x_i. Assuming weights and inputs are independent and zero-mean, Var(w_i x_i) = σ² × 1 = σ², so the variance of the sum is fan_in × σ² = 256σ². To keep the pre-activation variance equal to 1 (so signals don't explode), set σ² = 1/fan_in = 1/256 — this is exactly the LeCun initialization formula.

import numpy as np
fan_in = 256
sigma_sq = 1.0  # variance of each weight
# Var(pre-activation) = fan_in * sigma_sq
print(fan_in * sigma_sq)  # = 256 (explodes!)
# LeCun fix: sigma_sq = 1/fan_in
sigma_sq_lecun = 1 / fan_in
print(fan_in * sigma_sq_lecun)  # = 1.0 (stable)

The dying ReLU problem occurs when a neuron's pre-activation is always negative, making ReLU output permanently zero and the gradient permanently zero — the neuron never recovers. Large negative initial weights can push many neurons into this dead region right from the start, before any useful learning has occurred. He initialization reduces this risk by keeping pre-activations symmetrically distributed around zero with appropriate spread.

The fan_in for a convolutional layer equals k × k × C_in — the total number of weight connections contributing to a single output feature map location. This is because each output activation is computed as a weighted sum over the k×k spatial region across all C_in input channels. He initialization then sets variance = 2 / (k × k × C_in) to maintain stable activation variance.

LSUV first applies orthogonal initialization to all layers, then runs a mini-batch of real training data through the network one layer at a time. After each layer's forward pass, it rescales that layer's weights until the output variance is exactly 1.0. It is data-driven because the rescaling depends on the actual data distribution, not just mathematical assumptions about activation functions.

Fixup enables training residual networks without any batch normalization by carefully initializing the weights of residual branches. It scales the weight of each residual branch by L^(-1/(2m-2)) — where L is the number of residual blocks and m the layers per block — and sets the last layer of every residual branch to zero. This keeps gradient norms stable regardless of network depth, enabling reliable training of hundreds-of-layers-deep networks.

With weights from N(0,1) and large fan_in, the pre-activations become large in magnitude, pushing tanh outputs toward ±1 — its saturation region. The effective variance of activations collapses to near zero because most values are pinned to ±1. During backpropagation, the tanh gradient at saturation is nearly zero, producing vanishing gradients in early layers — exactly the problem that Xavier/LeCun initialization (using variance ~1/fan_in) is designed to prevent.

Random initialization starts from a task-agnostic point in the loss landscape, requiring many gradient steps for the network to first find a useful region of parameter space. Better alternatives include pre-training on large datasets (transfer learning), meta-learning algorithms like MAML that learn a good initialization across tasks, and data-driven schemes like LSUV that tailor the initial scale to the actual training distribution.

The correct call is nn.init.kaiming_normal_(layer.weight, mode='fan_in', nonlinearity='relu'). The mode='fan_in' setting preserves variance in the forward pass, and nonlinearity='relu' applies the gain factor of sqrt(2) appropriate for ReLU. The bias is typically initialized to zero separately with nn.init.zeros_.

import torch.nn as nn

layer = nn.Linear(128, 64)
# He normal initialization:
nn.init.kaiming_normal_(layer.weight, mode='fan_in', nonlinearity='relu')
# Standard zero bias:
nn.init.zeros_(layer.bias)
print('Weight variance:', layer.weight.var().item())  # ≈ 2/128 = 0.015625

The initial weights set the starting point in the high-dimensional loss landscape, and gradient descent can only move downhill from there. A good initialization (like He or Xavier) places the starting point in a region with clear, informative gradients — avoiding flat plateaus and saddle-point-heavy zones — so the optimizer converges to a good local minimum efficiently. Poor initialization can strand training in a region with near-zero gradients or cause divergence, fundamentally changing the final model quality.