Activation functions

An activation function is a mathematical function applied to the weighted sum of a neuron's inputs to produce its output. Its main job is to introduce non-linearity, allowing the network to learn complex patterns rather than just linear relationships.

Without non-linearity, stacking many layers would still be equivalent to a single linear transformation, so the network could only learn straight-line (linear) relationships. Non-linear activations let the network approximate complex, curved decision boundaries and arbitrary functions.

The sigmoid function squashes any real-valued input into the range (0, 1), never quite reaching 0 or 1. This makes it useful for representing probabilities, especially in the output of a binary classifier.

import numpy as np
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

The sigmoid function is defined as sigma(x) = 1 / (1 + e^(-x)). As x grows large and positive the output approaches 1, and as x grows large and negative it approaches 0.

The sigmoid derivative is sigma(x)(1 - sigma(x)), which reaches its maximum value of 0.25 at x = 0 (where sigma(x) = 0.5). Because this maximum is small, gradients shrink quickly through layers, contributing to the vanishing gradient problem.

The vanishing gradient problem occurs when gradients become extremely small as they are backpropagated through many layers, so early layers barely update and learning stalls. Saturating activations like sigmoid and tanh worsen this because their derivatives are near zero for large positive or negative inputs.

The tanh function outputs values in the range (-1, 1). Because it is zero-centered, its outputs are balanced around zero, which often helps optimization converge faster than sigmoid.

import numpy as np
def tanh(x):
    return np.tanh(x)

Tanh is zero-centered, meaning its outputs range from -1 to 1 with an average near zero, while sigmoid outputs are always positive (0 to 1). Zero-centered outputs make gradient updates more balanced and typically speed up convergence compared to sigmoid.

Tanh is a rescaled and shifted version of sigmoid: tanh(x) = 2*sigmoid(2x) - 1. This is why tanh has the same S-shape but is centered at zero with a range of (-1, 1) instead of (0, 1).

ReLU is defined as f(x) = max(0, x), meaning it outputs the input directly if it is positive and outputs 0 otherwise. It is the most widely used activation in hidden layers of deep networks due to its simplicity and effectiveness.

import numpy as np
def relu(x):
    return np.maximum(0, x)

ReLU outputs values in the range [0, infinity). Any negative input becomes 0, while positive inputs pass through unchanged, so the output is never negative.

ReLU only requires a simple comparison and selection (max with 0), whereas sigmoid and tanh involve costly exponential calculations. This makes ReLU much faster to compute and to differentiate, speeding up training of large networks.

The dying ReLU problem happens when neurons get stuck always outputting 0 because their inputs are consistently negative, making their gradient 0 so they never update. Such 'dead' neurons stop contributing to learning; variants like Leaky ReLU help avoid this.

The derivative of ReLU is 1 for positive inputs and 0 for negative inputs (it is technically undefined at exactly 0, but is usually set to 0 or 1 in practice). The constant gradient of 1 for positive values helps reduce the vanishing gradient problem.

Leaky ReLU allows a small, non-zero output for negative inputs, typically f(x) = x for x > 0 and f(x) = 0.01x for x <= 0. This small slope keeps the gradient non-zero for negative inputs, preventing neurons from completely dying.

import numpy as np
def leaky_relu(x, alpha=0.01):
    return np.where(x > 0, x, alpha * x)

Softmax converts a vector of raw scores (logits) into a probability distribution where each value is between 0 and 1 and all values sum to 1. It is used in the output layer of multi-class classification networks to represent the probability of each class.

import numpy as np
def softmax(x):
    e = np.exp(x - np.max(x))
    return e / e.sum()

The outputs of softmax are all non-negative and always sum to exactly 1, forming a valid probability distribution. This lets us interpret each output as the model's predicted probability for that class.

Sigmoid is typically used for binary classification because it outputs a single value between 0 and 1, interpreted as the probability of the positive class. A threshold (commonly 0.5) is then used to decide the predicted class.

Softmax models the classes as mutually exclusive by making the probabilities compete and sum to 1, which fits single-label problems where exactly one class is correct. Applying independent sigmoids would let probabilities vary independently and not sum to 1, which suits multi-label rather than single-label tasks.

The entire network would collapse into a single linear transformation, no matter how many layers it has, because a composition of linear functions is still linear. It could therefore only learn linear relationships, defeating the purpose of having multiple layers.

The step function outputs 1 if the input is above a threshold (e.g., 0) and 0 otherwise, giving a hard binary decision. It is rarely used in modern networks because its derivative is 0 almost everywhere, making gradient-based learning impossible.

Saturation means the function flattens out for large positive or negative inputs, so its output barely changes and its derivative approaches zero. Saturating activations like sigmoid and tanh slow learning because the near-zero gradients in their flat regions cause vanishing gradients.

Swish is defined as f(x) = x * sigmoid(x), making it a smooth, non-monotonic function. Unlike ReLU, it allows small negative outputs and has a smooth gradient everywhere, which can improve performance in some deep networks.

ELU outputs smooth negative values for negative inputs (using an exponential curve) instead of flat zero, which helps push the mean activation closer to zero. This can speed up learning and reduces the dying neuron problem seen with ReLU.

ReLU (and its variants) is the most common choice for hidden layers because it is fast to compute and helps avoid vanishing gradients. The output layer then uses softmax (multi-class) or sigmoid (binary) depending on the task.

During backpropagation, the chain rule multiplies the derivative of each layer's activation function into the gradient. If these derivatives are very small (as in saturated sigmoids), gradients vanish; if they are stable near 1 (as in ReLU for positive inputs), gradients flow more effectively.

Sigmoid(0) = 1 / (1 + e^0) = 1 / (1 + 1) = 0.5. This is the midpoint of the sigmoid curve, which is why 0.5 is the natural decision threshold in binary classification.

ReLU replaces every negative value with 0 and keeps positive values, giving [0, 0, 2, 0, 5]. Zero stays as 0 since max(0, 0) = 0.

Exponentiating large logits in softmax can cause overflow (numbers too big to store). A common fix is to subtract the maximum logit from all values before exponentiating, which leaves the result unchanged mathematically but keeps the exponentials small and stable.

import numpy as np
def stable_softmax(x):
    e = np.exp(x - np.max(x))
    return e / e.sum()

Parametric ReLU is like Leaky ReLU but the slope for negative inputs is a learnable parameter instead of a fixed constant. The network learns the best negative slope during training, giving more flexibility than a hand-picked value.

Softplus is defined as f(x) = ln(1 + e^x) and is a smooth, differentiable approximation of ReLU. Unlike ReLU, it has a continuous gradient everywhere, but it is more expensive to compute because of the logarithm and exponential.

A linear (identity) activation is usually used in the output layer for regression, because the target can be any real number and should not be squashed into a fixed range. Hidden layers may still use ReLU, but bounding the final output would distort the predictions.

GELU weights each input by the probability that a standard normal variable is less than it, roughly multiplying the input by a smooth gate based on its value. It behaves similarly to a smooth ReLU/Swish and is the default activation in many modern transformer models due to strong empirical performance.

ReLU is a sensible default for hidden layers because it is computationally cheap, trains quickly, and largely avoids vanishing gradients for positive inputs. If dying neurons become a problem, switching to Leaky ReLU, ELU, or GELU is a common next step.