Backpropagation explained

Backpropagation is an algorithm for efficiently computing the gradients of the loss function with respect to every weight and bias in the network. It works by applying the chain rule of calculus backward from the output layer to the input layer, reusing intermediate results. These gradients are then used by an optimizer (like gradient descent) to update the parameters and reduce the loss.

A neural network is a composition of many functions (layer after layer), and the loss depends on early weights only through all the later layers. The chain rule lets us compute the derivative of such a composite function by multiplying the local derivatives along the path. This is exactly what backpropagation does, multiplying layer-by-layer derivatives from output back to input.

First comes the forward pass, where inputs flow through the network to produce an output and a loss value, and intermediate activations are stored. Then comes the backward pass, where gradients are computed from the loss backward through each layer using the chain rule. After both passes, an optimizer updates the weights.

The gradient of the loss with respect to each weight depends on the inputs to that layer (the activations from the previous layer) and on local derivatives evaluated at the activation values. These quantities are produced during the forward pass, so they must be cached. Without them, the backward pass cannot evaluate the chain-rule products numerically.

Backpropagation computes the gradients (the direction of steepest increase of the loss), while gradient descent uses those gradients to actually update the weights by stepping in the opposite direction. They are separate but complementary: backpropagation provides the 'what direction', and gradient descent provides the 'take a step'. You need both to train the network.

Here w is the weight being updated, η (eta) is the learning rate controlling step size, and ∂L/∂w is the gradient of the loss with respect to that weight (computed by backpropagation). The minus sign means we move opposite to the gradient to decrease the loss. Backpropagation's job is to supply the ∂L/∂w term.

The loss function measures how far the network's predictions are from the true targets, producing a single number to minimize. Backpropagation starts from the derivative of this loss with respect to the network's output, then propagates that signal backward. Without a differentiable loss, there would be no starting gradient to back-propagate.

Many weights share common upstream factors (the gradient flowing into a layer is reused for every weight in that layer and passed further back). Backpropagation computes each layer's gradient signal once and reuses it, which is a form of dynamic programming. This makes computing all gradients roughly as cheap as one forward pass, rather than exponentially expensive.

By the chain rule, ∂L/∂w = (∂L/∂a)·(∂a/∂z)·(∂z/∂w). Since a = σ(z), ∂a/∂z = σ'(z), and since z = w·x + b, ∂z/∂w = x. Therefore ∂L/∂w = (∂L/∂a)·σ'(z)·x.

z = w*x + b
a = sigmoid(z)
# dL/dw = dL/da * sigmoid'(z) * x
dL_dw = dL_da * sigmoid_deriv(z) * x

The local gradient is the derivative of a node's output with respect to its inputs, computed using only that node's own operation (for example, the derivative of a multiply, add, or activation). During the backward pass, each node multiplies the incoming upstream gradient by its local gradient to produce the gradient passed further back. This modular view lets complex networks be differentiated piece by piece.

A computational graph represents a function as nodes (operations) connected by edges (data flow), breaking a complex expression into simple steps. Backpropagation traverses this graph in reverse, applying the chain rule at each node using its local gradient. This structure is what lets autodiff frameworks compute gradients automatically for arbitrary networks.

For addition, the local derivatives ∂z/∂x and ∂z/∂y are both 1. So the upstream gradient ∂L/∂z is passed unchanged to both inputs: ∂L/∂x = ∂L/∂z and ∂L/∂y = ∂L/∂z. Addition acts as a gradient 'distributor' that copies the gradient to all its inputs.

For multiplication, ∂z/∂x = y and ∂z/∂y = x, so each input's gradient is the upstream gradient multiplied by the OTHER input's value. Thus ∂L/∂x = (∂L/∂z)·y and ∂L/∂y = (∂L/∂z)·x. A multiply gate scales each branch by the other branch's forward value.

The derivative is σ'(z) = σ(z)·(1 − σ(z)). It is convenient because it is expressed entirely in terms of the already-computed forward output σ(z), so no extra exponentials are needed during the backward pass. This reuse of the forward activation speeds up backpropagation.

def sigmoid(z):
    return 1/(1+np.exp(-z))

def sigmoid_deriv(z):
    s = sigmoid(z)
    return s*(1-s)

ReLU(z) = max(0, z) has derivative 1 for z > 0 and 0 for z < 0 (undefined at 0, usually set to 0). During backpropagation, gradients pass through unchanged where the input was positive and are zeroed out where it was negative. This 'on/off gate' behavior is why ReLU can cause 'dead neurons' but also avoids shrinking gradients for active units.

In deep networks, gradients are products of many layer-wise derivatives; if these factors are consistently less than 1 (as with saturated sigmoid/tanh), their product shrinks exponentially toward zero as it propagates backward. The early layers then receive almost no gradient and learn extremely slowly or not at all. This is a key reason ReLU and careful initialization became popular.

Exploding gradients occur when the products of layer derivatives are consistently greater than 1, causing gradients to grow exponentially large during backpropagation, which makes weight updates huge and unstable (often producing NaNs). A common remedy is gradient clipping, which caps the gradient's norm at a maximum value before the update. Better initialization and normalization also help.

When a sigmoid's input is very large or very small, its output saturates near 1 or 0, and its derivative σ(z)(1−σ(z)) becomes nearly zero. Multiplying by this tiny local gradient kills the backward signal at that neuron, so its incoming weights barely update. Many saturated units across layers compound into the vanishing-gradient effect.

It computes derivatives by traversing the computational graph from outputs back to inputs (reverse order), accumulating gradients as it goes. Reverse mode is efficient when there are many inputs (weights) and few outputs (a single scalar loss), which is exactly the neural-network case. Computing the gradient for all parameters costs only a small constant factor more than one forward pass.

When a parameter (or activation) affects the loss along several paths in the graph, the multivariable chain rule says the total gradient is the SUM of the gradients contributed by each path. Backpropagation accumulates (adds) the incoming gradients at such a node before passing the result further back. Forgetting to sum these contributions is a common bug.

For MSE loss L = ½(a − y)² with output activation a = σ(z), the output-layer delta is δ = (∂L/∂a)·σ'(z) = (a − y)·σ'(z). This delta is the error signal that gets propagated backward to compute weight gradients. Each preceding layer's delta is derived from the next layer's delta times the connecting weights and local derivative.

If weights are all initialized to the same value, every neuron in a layer computes identical outputs and receives identical gradients, so they never differentiate (the symmetry problem). If weights are too large or too small, gradients can explode or vanish. Schemes like Xavier/He initialization keep the variance of activations and gradients stable across layers, helping backprop converge.

With identical (zero) weights, all neurons in a layer produce the same output and therefore receive the same gradient, so they update identically and remain indistinguishable forever. This 'symmetry' means the layer effectively behaves like a single neuron, no matter how many units it has. Random initialization breaks this symmetry so neurons can learn different features.

Backpropagation produces the gradient direction and magnitude, and the learning rate η scales how big a step is taken along the negative gradient. Too large an η can overshoot minima or diverge; too small an η makes training painfully slow. The learning rate does not change the gradient itself, only the size of the update applied.

In batch GD, gradients are computed over the entire dataset before one weight update; in stochastic GD (SGD), backprop runs and weights update after each single example; mini-batch GD uses a small group of examples per update, balancing the two. Mini-batch is most common because it gives stable gradient estimates while still updating frequently and exploiting vectorized hardware.

Each activation contributes its derivative as a factor in the chain-rule product, so its shape directly controls how gradients are scaled as they flow backward. Functions that saturate (sigmoid/tanh) shrink gradients and can cause vanishing gradients, while ReLU-like functions keep gradients near 1 for active units. The activation choice therefore strongly affects whether deep networks train successfully.

Gradient checking numerically estimates each gradient using finite differences, (L(w+ε) − L(w−ε)) / (2ε), and compares it to the analytic gradient produced by backpropagation. If the two match closely (small relative error), the backprop code is likely correct. It is too slow for training but invaluable for debugging custom implementations.

# Numerical gradient for one parameter
eps = 1e-5
num_grad = (loss(w+eps) - loss(w-eps)) / (2*eps)
# Compare with analytic backprop gradient
rel_error = abs(num_grad - backprop_grad) / (abs(num_grad)+abs(backprop_grad))

Finite differences require at least one extra forward pass per parameter (perturbing each weight separately), so cost scales with the number of parameters—millions of forward passes for a real network. Backpropagation computes the gradient for ALL parameters in a single backward pass, roughly one forward-pass worth of work. This efficiency is what makes training large networks feasible.

The delta of a hidden layer is obtained by taking the next layer's delta, multiplying it by the transpose of the weights connecting the two layers (to route the error backward), and then multiplying element-wise by the local activation derivative of the hidden layer. Symbolically, δ_l = (W_{l+1}ᵀ · δ_{l+1}) ⊙ f'(z_l). This recursive formula is the heart of backpropagation through layers.

The gradient of the loss with respect to a layer's weight matrix is the outer product of that layer's delta with the activations coming into the layer: ∂L/∂W_l = δ_l · a_{l-1}ᵀ. The bias gradient is simply the delta itself, ∂L/∂b_l = δ_l. These formulas turn the propagated error into concrete parameter updates.

Because backprop relies on the chain rule, every operation from inputs to loss must have a (sub)gradient so that local derivatives exist. Hard, non-differentiable steps like argmax or thresholding block the gradient flow. This is why we use smooth or piecewise-differentiable activations (sigmoid, ReLU) and softmax instead of a hard max during training.

As you run the forward pass, the framework records each operation onto a dynamic computational graph and remembers each op's local gradient rule. Calling loss.backward() then traverses that graph in reverse, applying the chain rule to populate the .grad of every parameter (autograd). The programmer only writes the forward pass; the backward pass is generated automatically.

import torch
x = torch.tensor(2.0, requires_grad=True)
y = x**2 + 3*x
y.backward()      # autograd runs backpropagation
print(x.grad)     # dy/dx = 2x + 3 = 7

By default, PyTorch ACCUMULATES (adds) new gradients into the existing .grad buffers rather than overwriting them, which supports advanced use cases. If you forget to call optimizer.zero_grad() before backward(), gradients from previous batches pile up and corrupt the current update. Zeroing ensures each step uses only the current batch's gradient.

for x, y in loader:
    optimizer.zero_grad()   # clear old gradients
    loss = criterion(model(x), y)
    loss.backward()         # backprop
    optimizer.step()        # update weights

The standard order is: (1) zero the gradients, (2) forward pass to compute predictions and loss, (3) backward pass (loss.backward()) to compute gradients via backpropagation, and (4) optimizer step to update the weights. Doing backward before the forward pass, or stepping before backward, would use stale or missing gradients and break training.