Loss functions explained

A loss function measures how far a model's prediction is from the true target value for a given example, producing a single number where lower is better. During training, the model adjusts its parameters to minimize this loss, usually via gradient descent. It is the quantity that defines what 'good' means for the model, so choosing the right loss directly shapes what the model learns.

The loss function typically refers to the error computed for a single training example, while the cost function (or objective function) refers to the average loss over the entire dataset or a batch. In practice the terms are often used interchangeably, but the cost is what is actually minimized over all data. Understanding this helps clarify that training minimizes an average, not the error on one point.

MSE is (1/n) * Σ (yᵢ - ŷᵢ)², where n is the number of examples, yᵢ is the true value, and ŷᵢ is the predicted value. Each squared difference penalizes the gap between prediction and truth, and the average gives one overall number. Because errors are squared, larger errors contribute disproportionately more to the total.

import numpy as np
mse = np.mean((y_true - y_pred) ** 2)

Because the differences are squared, an error of 4 contributes 16 while an error of 2 contributes only 4, so the penalty grows quadratically with the size of the error. This makes MSE very sensitive to outliers, since a few large mistakes can dominate the total loss. As a result, models trained on MSE try hard to avoid big errors even at the cost of small ones.

MAE is (1/n) * Σ |yᵢ - ŷᵢ|, the average of the absolute differences between predictions and true values. Because it does not square the errors, large errors grow only linearly, making MAE more robust to outliers than MSE. The trade-off is that MAE's gradient is constant in magnitude, which can make optimization near the minimum less smooth.

import numpy as np
mae = np.mean(np.abs(y_true - y_pred))

Huber loss behaves like MSE (squared) for small errors below a threshold δ and like MAE (linear) for large errors above δ. This makes it smooth and easy to optimize near zero while staying robust to outliers for large errors. It combines the best of both: sensitivity for small residuals and robustness for big ones.

MSE treats class labels as numbers and measures distance, but class labels are categories, so the squared distance between predicted probabilities and labels does not align well with classification goals. It also produces weak gradients when used with sigmoid outputs (the vanishing gradient problem), slowing learning. Cross-entropy loss is preferred because it strongly penalizes confident wrong predictions and gives healthier gradients.

For a true label y in {0,1} and predicted probability p, binary cross-entropy is -(y·log(p) + (1-y)·log(1-p)). When the prediction matches the label with high confidence, the loss is near zero, but a confident wrong prediction (e.g. p near 0 when y=1) drives the loss toward infinity. This heavy penalty encourages the model to be both correct and well-calibrated.

import numpy as np
bce = -np.mean(y*np.log(p) + (1-y)*np.log(1-p))

The term -log(p) blows up toward positive infinity as p approaches 0, so the loss becomes very large. This means a confident but wrong prediction is penalized extremely heavily. This property pushes the model away from making overconfident mistakes.

Categorical cross-entropy is -Σ yᵢ·log(pᵢ) summed over all classes, where y is a one-hot true vector and p is the predicted probability distribution. It is paired with a softmax output layer that converts raw scores (logits) into a probability distribution over classes. Because the label is one-hot, the loss reduces to -log(probability assigned to the correct class).

Softmax converts a vector of raw scores (logits) into a probability distribution by exponentiating each value and normalizing so they sum to 1. It is combined with cross-entropy because together they produce a clean, well-behaved gradient (predicted probability minus the true label) that makes training stable and efficient. This pairing is the standard setup for multi-class classification.

import numpy as np
def softmax(z):
    e = np.exp(z - np.max(z))
    return e / e.sum()

Cross-entropy comes from information theory, where -log(p) measures the 'surprise' of an outcome that the model assigned probability p. If the model assigns high probability to the true outcome, the surprise (and loss) is low; if it assigns low probability, the surprise is high. Minimizing cross-entropy thus means making the model least surprised by the true labels.

Minimizing cross-entropy is mathematically equivalent to maximizing the likelihood of the observed labels under the model (maximum likelihood estimation). The negative log-likelihood of the data is exactly the cross-entropy loss summed over examples. So a model trained with cross-entropy is finding parameters that make the actual data most probable.

Hinge loss is max(0, 1 - y·ŷ), where y is the true label in {-1, +1} and ŷ is the raw model score. It is the loss used by Support Vector Machines (SVMs) and pushes correct predictions to have a margin of at least 1, giving zero loss only when the prediction is correct and confident enough. Predictions inside the margin or on the wrong side incur a linearly increasing penalty.

Gradient descent updates parameters using the gradient (derivative) of the loss with respect to those parameters, so the loss must be differentiable to provide that gradient. If the loss is flat or non-differentiable everywhere useful, there is no gradient signal to guide the updates. This is why smooth surrogate losses are often used instead of non-differentiable metrics like accuracy.

Accuracy is a step-like function that stays flat as predictions change slightly and only jumps when a prediction flips classes, so its gradient is zero almost everywhere. This gives no useful direction for gradient descent to follow. Instead, smooth surrogate losses like cross-entropy are minimized, and accuracy is tracked separately as an evaluation metric.

A loss function is what the model minimizes during training and must be differentiable for gradient descent, while an evaluation metric (like accuracy, F1, or RMSE) is what humans use to judge model quality and need not be differentiable. Sometimes they coincide, but often we optimize a smooth surrogate loss while reporting a more interpretable metric. They serve different purposes: optimization versus assessment.

A convex loss function has a single global minimum and no other local minima, so its graph curves upward like a bowl. Convexity is desirable because gradient descent is guaranteed to reach the global minimum regardless of the starting point. Losses like MSE for linear regression are convex, whereas deep neural network losses are generally non-convex.

The algorithm computes the gradient of the loss with respect to each parameter, indicating the direction of steepest increase. Each parameter is then nudged in the opposite direction (downhill) by a step proportional to the learning rate: w := w - learning_rate · ∂Loss/∂w. Repeating this gradually reduces the loss toward a minimum.

# one gradient descent step
w = w - learning_rate * grad_loss_w

Regularization adds a penalty term to the loss based on the size of the model's weights, so the total loss becomes the data loss plus a weighted penalty. This discourages overly large weights and overly complex models, helping the model generalize rather than memorize the training data. Common forms are L2 (sum of squared weights) and L1 (sum of absolute weights).

# L2-regularized loss
total = data_loss + lam * np.sum(w**2)

L2 regularization penalizes the sum of squared weights, shrinking all weights smoothly toward (but not exactly to) zero. L1 regularization penalizes the sum of absolute weights and tends to drive some weights exactly to zero, producing sparse models that effectively perform feature selection. Thus L1 encourages sparsity while L2 encourages small, distributed weights.

RMSE is the square root of MSE: √[(1/n)·Σ(yᵢ - ŷᵢ)²]. Taking the square root returns the error to the same units as the target variable, making it easier to interpret. It preserves MSE's sensitivity to large errors while being more directly readable.

import numpy as np
rmse = np.sqrt(np.mean((y_true - y_pred)**2))

If the predicted probability p is exactly 0 or 1, then log(0) is undefined (negative infinity), causing numerical errors. Libraries handle this by clipping p to a small range like [ε, 1-ε], or by combining the softmax and log into a single numerically stable operation (log-softmax / 'from_logits'). This keeps the loss finite and the gradients well-behaved.

import numpy as np
p = np.clip(p, 1e-12, 1 - 1e-12)
loss = -np.mean(y*np.log(p) + (1-y)*np.log(1-p))

KL divergence measures how much one probability distribution differs from a reference distribution, and it is always non-negative, equaling zero only when the distributions match. Cross-entropy equals the entropy of the true distribution plus the KL divergence from the true to the predicted distribution. Since the true distribution's entropy is fixed, minimizing cross-entropy is equivalent to minimizing the KL divergence.

Regression tasks predict continuous numbers, so they use losses like MSE, MAE, or Huber that measure numeric distance. Classification tasks predict categories or probabilities, so they use losses like binary or categorical cross-entropy (or hinge loss) that measure how well predicted probabilities match the true class. Matching the loss to the task type is essential for effective training.

Focal loss modifies cross-entropy by adding a factor (1 - p)^γ that down-weights the loss from easy, well-classified examples so the model focuses on hard, misclassified ones. It was designed to address severe class imbalance, such as object detection where background examples vastly outnumber objects. By reducing the contribution of abundant easy examples, it prevents them from overwhelming the gradient.

Class weighting multiplies the loss contribution of each example by a weight that is higher for rare (minority) classes and lower for common ones. This forces the model to pay proportionally more attention to underrepresented classes rather than always predicting the majority class. It is a simple way to counteract bias caused by imbalanced data without changing the data itself.

Label smoothing replaces hard one-hot targets (like 1 and 0) with slightly softened values (like 0.9 and a small share spread across the rest). This discourages the model from becoming overconfident and improves calibration and generalization. It effectively tells the model that the correct class is very likely but not absolutely certain.

For the per-example loss (ŷ - y)², the derivative with respect to ŷ is 2(ŷ - y). This shows the gradient is proportional to the error, so larger errors produce larger updates. The factor of 2 is often absorbed into the learning rate or removed by defining the loss with a 1/2 factor.

When softmax and cross-entropy are combined, the gradient of the loss with respect to each logit simplifies to (predicted probability - true label), i.e. (p - y). This elegant form avoids the exploding or vanishing terms that arise if the two operations are differentiated separately, making backpropagation efficient and numerically stable. It is one reason this pairing is the default for classification.

A robust loss is one whose value (and gradient) is not dominated by a few extreme, unusual data points. MAE is more robust than MSE because it grows linearly with error rather than quadratically, so a single huge error does not overwhelm the total. This makes MAE or Huber loss preferable when the data contains noisy outliers.

The mean of the target values minimizes MSE, because MSE is minimized at the average of the data. This is a key property: predicting the mean gives the lowest possible squared-error loss for a constant predictor. In contrast, the median minimizes MAE, not MSE.

The median of the target values minimizes MAE, whereas the mean minimizes MSE. This is why MAE is more robust to outliers: the median is far less affected by extreme values than the mean. Knowing this helps explain the different behavior of models trained with these two losses.

The loss function defines exactly what the model optimizes, so a mismatched loss makes the model 'good' at the wrong objective. For example, using MSE for classification gives weak gradients and poor probability estimates, while ignoring class imbalance in the loss can make a model that just predicts the majority class. Even with perfect data and a strong architecture, optimizing the wrong target leads to disappointing real-world results.