Gradient clipping

Gradient clipping is a technique that prevents exploding gradients by capping or rescaling gradient values when their magnitude exceeds a set threshold during training. It is applied after backpropagation computes the gradients and before the optimizer uses them to update weights. Without it, very large gradients can cause parameter updates to be so large that training becomes unstable or diverges.

During backpropagation, gradients are computed by multiplying together many partial derivatives from each layer using the chain rule. If the weight matrices have eigenvalues slightly greater than 1, these repeated multiplications cause gradient magnitudes to grow exponentially as they flow backward through many layers, a phenomenon called exploding gradients.

Gradient clipping directly addresses the exploding gradient problem, where gradients become extremely large during backpropagation and cause dangerously large parameter updates. It does not fix the vanishing gradient problem, where gradients shrink toward zero, which is a separate issue addressed by architectures like LSTMs and residual connections.

The vanishing gradient problem occurs when gradients become extremely small during backpropagation through many layers, causing early layers to learn very slowly or stop learning. Gradient clipping only reduces large gradients; it cannot amplify tiny ones. Solutions for vanishing gradients include LSTMs, residual connections, and careful weight initialization.

Gradient norm clipping computes the L2 norm of all gradients combined and scales all of them down by the same factor if the norm exceeds the threshold, preserving the gradient direction. Gradient value clipping independently clips each gradient element to a fixed range [-clip_value, +clip_value], which can alter the overall gradient direction since different components are treated separately.

Gradient value clipping clips each individual gradient element to the range [-clip_value, +clip_value] independently of the other elements. Its main drawback is that components may be clipped by different amounts, which changes the relative proportions between gradient components and therefore distorts the direction of the gradient vector. Norm clipping avoids this by scaling all gradients by the same factor.

The L2 norm of a gradient vector g with elements g₁, g₂, ..., gₙ is computed as ||g||₂ = √(g₁² + g₂² + ... + gₙ²), which is the square root of the sum of squares of all elements. For a full neural network, this is calculated across the gradient values of every single trainable parameter combined into one large vector.

torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm) applies gradient norm clipping in PyTorch. The first argument is an iterable of parameters whose gradients will be clipped, and max_norm is the maximum allowed L2 norm of the combined gradient. If the computed norm exceeds max_norm, all gradients are scaled down so the total norm equals exactly max_norm.

torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

Recurrent Neural Networks (RNNs) most commonly require gradient clipping because during backpropagation through time (BPTT), the same weight matrix is multiplied at every time step. For long sequences, this repeated multiplication can cause the gradient norm to grow exponentially with sequence length, making gradient clipping essential for stable RNN training.

BPTT is the algorithm used to train RNNs by unrolling the network across all time steps and applying standard backpropagation on this unrolled computational graph. Because the same weight matrix is multiplied at each time step, the gradient signal can either grow exponentially (exploding gradients, when eigenvalues exceed 1) or shrink exponentially (vanishing gradients, when eigenvalues are less than 1) with increasing sequence length.

Symptoms of exploding gradients include the training loss suddenly jumping to very large values or becoming NaN (Not a Number), model weights growing to extreme magnitudes, and training accuracy collapsing after initial progress. Monitoring the gradient norm at each step is the most reliable way to confirm and quantify gradient explosion before it causes these visible failures.

Gradient clipping must be placed after loss.backward() (which computes and stores gradients in each parameter's .grad attribute) and before optimizer.step() (which reads those gradients to update weights). Clipping before backward() means there are no gradients yet; clipping after step() means the optimizer already used the uncapped gradients, making clipping useless.

optimizer.zero_grad()
loss = criterion(model(x), y)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()

max_norm is the maximum allowed value for the total L2 norm of all gradients combined. If the computed gradient norm is at or below max_norm, gradients are left unchanged. If it exceeds max_norm, all gradients are multiplied by (max_norm / computed_norm), rescaling the entire gradient vector so its L2 norm becomes exactly max_norm.

No, gradient norm clipping does not change the gradient direction. It multiplies every gradient element by the same scalar factor (max_norm / current_norm), which is equivalent to scaling the gradient vector without rotating it. The relative proportions between all gradient components remain identical, so the optimization step moves in exactly the same direction, just with a smaller magnitude.

No, gradient value clipping does not preserve the gradient direction. Because each gradient element is clipped independently to [-clip_value, +clip_value], components that are large get truncated while small components are left untouched. This changes the relative magnitudes of different components, which alters the overall direction of the gradient vector in parameter space.

A max_norm of 1.0 is a conservative yet effective choice that prevents catastrophically large gradient updates while still allowing the model to take meaningful learning steps. It is empirically found to work well across many NLP and sequence modeling tasks. Ideally, the best threshold should be found by monitoring gradient norms early in training to understand the typical unclipped norm for a specific task.

The clipped gradient g' is computed as g' = g × min(1, θ / ||g||₂), where g is the original gradient vector, ||g||₂ is its L2 norm, and θ is the clipping threshold. When ||g||₂ ≤ θ, the factor equals 1 and the gradient is unchanged. When ||g||₂ > θ, all gradients are scaled by θ / ||g||₂, reducing the norm to exactly θ while preserving direction.

clip_value sets the absolute boundary for each individual gradient element: every gradient value is clamped to the range [-clip_value, +clip_value]. For example, with clip_value=0.5, a gradient element of 2.3 is clipped to 0.5, and a gradient of -1.7 is clipped to -0.5. Elements already within the range are left unchanged.

torch.nn.utils.clip_grad_value_(model.parameters(), clip_value=0.5)

The effective maximum parameter update per step is approximately clip_threshold × learning_rate. Gradient clipping bounds the gradient norm to at most clip_threshold, and the learning rate then scales this bounded gradient into a weight change. If either value is set too large, updates can still be destabilizing; tuning both together is important for stable and efficient training.

clip_grad_norm_() returns the total L2 norm of all the gradients before any clipping is applied. Logging this value during training is useful because if the pre-clip norm is frequently much larger than max_norm, it signals that gradients are regularly exploding. Tracking this over training steps helps practitioners detect instability early and tune the clipping threshold appropriately.

The return value of clip_grad_norm_() is the pre-clipping L2 norm and can be logged at each training step. If this value is regularly much greater than max_norm, clipping is frequently triggering, indicating potential instability. Tracking it with a tool like TensorBoard over many steps provides a clear picture of gradient health throughout training.

grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
writer.add_scalar('train/grad_norm', grad_norm, global_step)

Global norm clipping computes a single L2 norm across all parameters in the entire network and rescales every gradient by the same factor. Per-layer norm clipping computes a separate norm for each layer and clips each layer independently. Global norm clipping is generally preferred because it preserves the relative gradient magnitudes across layers and is the behavior implemented by PyTorch's clip_grad_norm_().

LSTMs use gating mechanisms (forget, input, output, and cell gates) that create more stable gradient pathways through time and help mitigate the vanishing gradient problem. However, LSTMs do not fully prevent exploding gradients, and gradient clipping is still commonly applied when training LSTMs on long sequences to handle occasional gradient spikes that can still destabilize training.

A gradient norm that is orders of magnitude larger than the clipping threshold (for example, norms of 500 or 10,000 when using max_norm=1.0) strongly indicates gradient explosion. The clearest and most extreme sign is when the training loss becomes NaN, which occurs when gradient values overflow floating-point precision during the parameter update step.

In a vanilla RNN, backpropagation multiplies the weight matrix (or its Jacobian) T times for a sequence of length T. If the largest eigenvalue of this matrix is λ > 1, the gradient norm grows proportionally to λᵀ, which is exponential in sequence length. This is why gradient explosion becomes dramatically more severe for longer input sequences and why gradient clipping is critical for RNNs trained on long sequences.

Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio formally analyzed the exploding and vanishing gradient problems in RNNs and proposed gradient norm clipping as a practical solution in their 2013 paper 'On the difficulty of training recurrent neural networks.' This paper provided the theoretical framework explaining why these problems occur and how clipping helps, and it remains a foundational reference in the field.

With a very small clipping threshold, gradients are almost always rescaled to a tiny value regardless of the true gradient magnitude. This severely slows learning because parameter updates become negligibly small in every step, effectively creating a situation similar to the vanishing gradient problem where the model makes almost no progress even if the true gradient signal is informative.

With a very large clipping threshold, the gradient norm almost never exceeds it, so clipping is effectively never triggered during training. The network trains as if no gradient clipping were applied, leaving it fully vulnerable to exploding gradients and the training instability that follows. A threshold that is too large provides no meaningful protection.

Yes, gradient clipping is standard practice in training large transformer models including GPT and BERT variants. Although transformers use layer normalization and attention mechanisms that provide some training stability, gradient clipping with max_norm=1.0 is routinely applied in official training scripts to handle occasional gradient spikes that can still destabilize training, especially early in the training process.

clip_grad_norm_(params, 1.0) computes the total L2 norm of ALL gradients and scales all of them down proportionally if the norm exceeds 1.0, preserving the gradient direction. clip_grad_value_(params, 1.0) clips every individual gradient element independently to [-1.0, 1.0], which can change the gradient direction because different components may be clipped by different amounts. They use the same threshold number but have very different geometric behaviors.

Gradient clipping constrains the maximum step size taken in parameter space at each training iteration, preventing large jumps that could cross sharp cliffs or highly curved regions of the loss surface and cause divergence. This keeps the optimization trajectory in a more controlled region of the landscape without changing the loss surface itself. The model still follows gradient descent in the correct direction, just with a bounded step magnitude.

The implementation computes the L2 norm by taking the square root of the sum of squared values across all gradient arrays. If this norm exceeds max_norm, all gradients are scaled by (max_norm / norm), which preserves their direction. A small epsilon prevents division by zero when all gradients are exactly zero.

import numpy as np

def clip_grad_norm(gradients, max_norm, eps=1e-6):
    # gradients: list of numpy arrays, one per parameter layer
    total_norm = np.sqrt(sum(np.sum(g ** 2) for g in gradients))
    clip_coef = max_norm / (total_norm + eps)
    if clip_coef < 1.0:
        gradients = [g * clip_coef for g in gradients]
    return gradients, total_norm

Adam adapts per-parameter learning rates based on gradient history using first and second moment estimates, which reduces sensitivity to gradient scale variations. However, Adam's adaptive rates do not prevent raw gradient magnitudes from being very large, especially early in training or during sudden data distribution shifts. Gradient clipping provides an additional safety layer that bounds the raw gradient feeding into Adam's update rule, preventing sudden destabilizing spikes.