Gradient vanishing and exploding

The vanishing gradient problem occurs when gradients become extremely small as they are backpropagated from the output layer toward the earlier layers. Because each layer's gradient is a product of many small factors, the signal shrinks toward zero, so early-layer weights barely update and learn very slowly. This makes deep networks hard to train, especially the layers closest to the input.

The exploding gradient problem happens when gradients grow very large during backpropagation, often because repeated multiplication of factors greater than one compounds the values. This causes huge, unstable weight updates that can make the loss oscillate or become NaN (not a number). It is common in deep networks and recurrent networks with poor initialization or large weights.

During backpropagation the gradient for an early layer is a product of many per-layer terms (weights and activation derivatives) via the chain rule. When you multiply many numbers that are mostly less than one, the product decays toward zero (vanishing); when they are mostly greater than one, it blows up (exploding). More layers means more factors in this product, amplifying both effects.

Backpropagation uses the chain rule, so the gradient of the loss with respect to an early weight is a product of derivatives across all later layers. If these derivative factors are consistently smaller than one their product approaches zero, and if they are consistently larger than one their product grows without bound. Thus the multiplicative structure of the chain rule is the mathematical root of both problems.

The sigmoid's derivative is sigma(x)(1 - sigma(x)), which has a maximum value of 0.25 at x = 0. Because every factor passed back through a sigmoid layer is at most 0.25, multiplying many such factors across deep layers drives the gradient rapidly toward zero. This is a key reason sigmoid activations cause vanishing gradients in deep networks.

import numpy as np
x = np.linspace(-10, 10, 1000)
s = 1/(1+np.exp(-x))
print(np.max(s*(1-s)))  # ~0.25

ReLU outputs the input directly for positive values, so its derivative is exactly 1 there instead of a small fraction. Since the gradient is multiplied by 1 (not by something like 0.25 as with sigmoid), the signal passes through positive units without shrinking. This lets gradients flow more easily through many layers, reducing vanishing.

A 'dead ReLU' is a neuron stuck outputting zero for all inputs because its weighted input is always negative, where ReLU's derivative is 0. With a zero gradient, that neuron's weights never update and it stops learning permanently. This is a form of vanishing gradient specific to ReLU units, often caused by large negative biases or too-high learning rates.

Leaky ReLU outputs a small negative slope (e.g., 0.01x) for negative inputs instead of a flat zero, so its derivative is a small positive number rather than 0 there. This keeps a nonzero gradient flowing even for negative-input neurons, preventing them from dying. As a result more neurons keep learning and gradient flow improves.

def leaky_relu(x, alpha=0.01):
    return x if x > 0 else alpha * x

Gradient clipping rescales or caps gradients when their norm exceeds a chosen threshold, so updates stay within a safe range. It primarily solves the exploding gradient problem by preventing huge, destabilizing weight updates that would otherwise cause the loss to diverge or become NaN. It is widely used when training recurrent neural networks.

import torch
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

A residual connection adds the input of a block directly to its output, so the layer learns a residual function instead of a full transformation. During backpropagation this creates a shortcut path where the gradient can flow through the identity term with a derivative of 1, bypassing the multiplicative shrinking of the stacked layers. This lets gradients reach early layers, enabling very deep networks to train.

def residual_block(x, layer):
    return x + layer(x)  # identity shortcut + transformation

Batch normalization normalizes each layer's inputs to have a stable mean and variance during training, keeping activations in a well-behaved range. This prevents activations from drifting into the saturated regions of functions like sigmoid/tanh (where derivatives are tiny) and keeps the scale of signals consistent across layers. The result is smoother, more stable gradient flow that resists both vanishing and exploding.

Xavier initialization sets the initial weight variance based on the number of input and output units so that the variance of activations and gradients stays roughly constant across layers. By keeping signals from shrinking or growing as they pass through layers, it helps prevent both vanishing and exploding gradients at the start of training. It is designed for activations like tanh and sigmoid.

import numpy as np
# Xavier/Glorot: variance uses both fan-in and fan-out
W = np.random.randn(n_in, n_out) * np.sqrt(2.0/(n_in + n_out))

He initialization scales the initial weight variance by 2/n_in instead of 1/n_in, accounting for the fact that ReLU zeroes out about half of its inputs. This extra factor compensates for the lost variance, keeping activation and gradient magnitudes stable through ReLU layers. It therefore reduces vanishing/exploding gradients better than Xavier when using ReLU.

import numpy as np
W = np.random.randn(n_in, n_out) * np.sqrt(2.0/n_in)  # He init

An RNN applies the same recurrent weight matrix repeatedly across time steps, so backpropagation through time multiplies that matrix's effect many times. If the relevant factor is below one the gradient vanishes over long sequences, and if it is above one it explodes. This makes plain RNNs struggle to learn long-range dependencies.

LSTMs introduce a cell state with a near-linear path controlled by gates (forget, input, output) that regulate what information is kept or discarded. The cell state allows gradients to flow across many time steps with little shrinking, because the forget gate can keep the path close to an identity (multiplying by ~1). This preserves long-range gradient signal far better than a plain RNN.

A telltale sign is the loss suddenly becoming NaN or infinity, or oscillating wildly instead of decreasing smoothly. You may also see the gradient norm or weight values shooting up to very large magnitudes. These instabilities indicate that updates have grown too large to control.

A common symptom is that the early layers' weights barely change while later layers update normally, and the loss plateaus or decreases extremely slowly. Monitoring gradient norms shows very small values for the earliest layers. The network seems to stall, learning little despite many epochs.

The weight update is the learning rate multiplied by the gradient, so a large learning rate magnifies already-large gradients into even bigger, destabilizing steps. A learning rate that is too high can therefore turn manageable gradients into divergence or NaN losses. Lowering the learning rate (alongside clipping) is a common remedy.

An activation is saturated when its input is so large in magnitude that the output is nearly flat (e.g., sigmoid near 0 or 1). In these flat regions the derivative is almost zero, so very little gradient passes backward through that neuron. When many neurons saturate, gradients to earlier layers vanish.

Tanh is zero-centered and has a maximum derivative of 1 (at 0), versus sigmoid's maximum of 0.25, so it passes stronger gradients near the origin. However, tanh still saturates for large positive or negative inputs, where its derivative approaches zero. Therefore tanh reduces but does not eliminate the vanishing gradient problem in deep networks.

The gradient is multiplied by 0.5 ten times, giving 0.5^10, which is about 0.001 of its original size. This roughly thousand-fold shrinkage illustrates how quickly gradients vanish with depth when per-layer factors are below one. After many more layers the gradient would be effectively zero.

print(0.5**10)  # ~0.000977

The gradient is multiplied by 2 ten times, giving 2^10 = 1024, so it grows roughly a thousand-fold. This rapid growth shows how factors above one cause exploding gradients with depth. Continuing deeper would quickly overflow into numerical infinity.

print(2**10)  # 1024

By computing the L2 norm of each layer's gradients during training, you can see how gradient magnitude changes across layers. Norms that shrink toward zero in early layers indicate vanishing gradients, while norms that blow up to very large values indicate exploding gradients. This per-layer view pinpoints where the gradient flow breaks down.

for name, p in model.named_parameters():
    if p.grad is not None:
        print(name, p.grad.norm().item())

Truncated BPTT limits how many time steps the gradient is propagated backward, splitting long sequences into shorter chunks for the backward pass. By shortening the chain of multiplications, it reduces the chance of gradients vanishing or exploding over very long sequences and saves memory. The trade-off is that the model cannot directly learn dependencies longer than the truncation window.

If initial weights are large, the activations and the per-layer Jacobian factors in backpropagation tend to be greater than one. Multiplying many such large factors across layers makes the gradient grow exponentially, producing exploding gradients from the very first updates. Proper variance-scaled initialization keeps these factors near one to avoid this.

A GRU uses update and reset gates to control how much past information is carried forward and how much is overwritten at each step. The update gate can keep the hidden state nearly unchanged, creating a near-identity path that lets gradients flow across time with little shrinkage. This mitigates vanishing gradients while using fewer gates than an LSTM.

Vanishing gradients hurt the earliest (input-side) layers the most because their gradients are products of the longest chains of small factors. Later layers near the output still receive sizable gradients and train normally. As a result, early layers learn slowly or not at all, limiting the network's ability to learn low-level features.

Repeated multiplication by the recurrent weight matrix is governed by its largest eigenvalue magnitude (spectral radius). If that radius is below 1, repeated products shrink and gradients tend to vanish over time; if it is above 1, they tend to grow and gradients explode. Keeping the spectral radius near 1 helps maintain stable gradient flow.

ELU and SELU output smooth negative values for negative inputs (rather than zero), which keeps mean activations closer to zero and avoids fully blocking gradients on negative-input neurons. SELU additionally has self-normalizing properties that keep activations at stable mean and variance across layers. Both effects reduce vanishing gradients compared with plain ReLU in deep networks.

Layer normalization normalizes across the features of a single example rather than across a batch, so it does not depend on batch size or batch statistics. This makes it well suited to RNNs and small batches, where batch statistics are unreliable, while still keeping activations and gradients in a stable range. The result is steadier gradient flow across time steps.

For an identity mapping y = x, the derivative dy/dx is exactly 1, so backpropagating through it multiplies the gradient by 1 and preserves its magnitude. Residual blocks add this identity path alongside the learned transformation, guaranteeing at least an unshrunk route for the gradient. This prevents the gradient from decaying to zero even in very deep stacks.

This pattern shows a vanishing gradient situation: the gradient magnitude shrinks dramatically from the output side to the input side. The early layers receive almost no learning signal, so they update extremely slowly even though later layers (and overall loss) improve. Remedies include better initialization, ReLU-type activations, batch/layer normalization, or residual connections.

Each sigmoid layer caps the gradient factor at 0.25 and saturates for large inputs, so backpropagating through many such layers multiplies many tiny numbers and the gradient vanishes. No optimizer can update weights effectively when the gradients reaching early layers are essentially zero. Structural fixes (ReLU, normalization, residual connections) are needed, not just a better optimizer.

Modern very deep networks typically combine ReLU-family activations, variance-scaled initialization (He/Xavier), normalization layers (batch or layer norm), and residual/skip connections. Together these keep per-layer gradient factors near one and provide shortcut paths, so gradients neither vanish nor explode. Gradient clipping is often added for recurrent or transformer training to control occasional spikes.