Skip to content
elevatedevco

Cheat sheet · Deep Learning

Activation & Loss Functions cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

Cheat sheet

A dense reference on activation functions (what introduces nonlinearity and shapes gradient flow) and loss functions (what the network actually optimizes), covering modern choices, the math, numerical stability, and the design tradeoffs interviewers probe.

Activation Functions: Core Families

  • Purpose: inject nonlinearity so stacked layers can approximate non-linear functions; without it a deep net of linear layers collapses to a single linear map (one matrix).
  • ReLU: f(x)=max(0,x). Cheap, induces sparsity, no positive-side saturation. Default for CNNs/MLPs. Risk: 'dying ReLU' (units stuck at 0 with zero gradient).
  • Leaky/Parametric ReLU: f(x)=x if x>0 else a*x (a fixed ~0.01 for Leaky, learned per-channel for PReLU). Allows a small negative slope so units can recover.
  • ELU/SELU: smooth negative branch (a*(e^x-1)); push mean activations toward zero. SELU enables self-normalizing nets under specific init/architecture.
  • GELU: x*Phi(x), where Phi is the standard Gaussian CDF. Smooth, probabilistically-gated; standard in Transformers (BERT, GPT, ViT). Exact form or a tanh approximation are both used.
  • SiLU/Swish: x*sigmoid(beta*x) (beta=1 for SiLU). Smooth, non-monotonic, self-gated; common in EfficientNet and many modern vision/LLM blocks. Mish (x*tanh(softplus(x))) is a similar smooth variant.
  • Sigmoid & Tanh: squash to (0,1)/(-1,1). Saturate at extremes (vanishing gradients); mostly relegated to gates (LSTM/GRU) and output heads. Softplus = log(1+e^x) is a smooth ReLU.
  • GLU variants (SwiGLU, GeGLU): split a projection into value and gate paths and multiply them. SwiGLU is the dominant FFN activation in modern LLMs (LLaMA, PaLM).

Choosing & Tuning Activations

  • Hidden layers: ReLU is a strong default; GELU/SiLU when smoothness helps optimization (Transformers, deep nets); SwiGLU for LLM FFNs.
  • Output layer is dictated by the task, NOT a free choice: linear for regression, sigmoid for binary/multi-label, softmax for single-label multiclass.
  • Zero-centered, smooth activations (GELU, tanh) ease gradient flow vs. ReLU's strictly non-negative output, but ReLU is faster and often good enough.
  • Init must match activation: He/Kaiming init for ReLU-family, Xavier/Glorot for tanh/sigmoid; mismatched init causes vanishing/exploding signals.
  • Watch saturation: sigmoid/tanh gradients vanish for large |x|; pair with normalization (BatchNorm/LayerNorm) to keep pre-activations centered.
  • Non-monotonic (Swish/GELU) and gated (SwiGLU) variants tend to win at scale but add compute; benchmark before adopting.

Softmax & the Logits Interface

  • Softmax: sigma(z)_i = e^{z_i} / sum_j e^{z_j}; maps logits to a probability simplex (non-negative, sums to 1). Used for single-label multiclass.
  • Temperature: softmax(z/T). T>1 softens (more uniform, used in distillation/sampling); T<1 sharpens toward argmax. Post-hoc temperature scaling improves calibration.
  • Numerical stability: subtract max(z) before exp to avoid overflow: softmax(z) = softmax(z - max z). Frameworks do this internally.
  • Prefer fused 'logits-in' losses (cross_entropy on raw logits) over softmax-then-log; they use log-sum-exp for stability and correct gradients.
  • Softmax is shift-invariant (adding a constant to all logits changes nothing) but NOT scale-invariant; scaling logits changes confidence/sharpness.
  • Multi-label (independent classes) uses per-logit sigmoid + BCE, not softmax; mixing these up is a classic bug.

Loss Functions: Regression & Classification

  • MSE (L2): mean((y-yhat)^2). Smooth, penalizes large errors quadratically; sensitive to outliers. Corresponds to Gaussian-noise MLE; optimizes the conditional mean.
  • MAE (L1): mean(|y-yhat|). Robust to outliers, constant-magnitude gradient; non-smooth at 0. Optimizes the conditional median.
  • Huber/Smooth-L1: quadratic for small residuals (|r|<delta), linear beyond; combines MSE smoothness with MAE robustness.
  • Binary cross-entropy (BCE): -[y*log p + (1-y)*log(1-p)] with sigmoid; for binary and multi-label (applied per class).
  • Categorical cross-entropy: -sum_i y_i*log p_i with softmax; for single-label multiclass. Equals the negative log-likelihood of the true class.
  • Cross-entropy = H(target) + KL(target||pred); with one-hot targets H(target)=0, so CE equals KL exactly and minimizing it matches the label distribution.
  • Label smoothing: replace hard targets with 1-eps+eps/K for the true class and eps/K for the rest; regularizes, improves calibration, prevents over-peaked softmax.

Specialized & Modern Losses

  • Focal loss: -(1-p_t)^gamma * log(p_t); down-weights easy examples to handle extreme class imbalance (object detection, rare events).
  • Dice / IoU loss: overlap-based, used in segmentation where pixel imbalance weakens CE; often combined (Dice+CE). Tversky generalizes Dice to weight FP vs FN.
  • Contrastive / Triplet / InfoNCE: distance/similarity losses for metric and self-supervised learning (SimCLR, CLIP use InfoNCE).
  • Hinge loss: max(0, 1 - y*yhat) with y in {-1,+1}; margin-based, classic SVM; rarely used in deep nets vs. cross-entropy.
  • Class weighting: scale per-class loss by inverse frequency to counter imbalance; an alternative/complement to focal loss.
  • LLM training: next-token cross-entropy for pretraining; preference/alignment losses (DPO, and reward-model + PPO for RLHF) for fine-tuning.
  • Auxiliary/regularization terms (L2 weight decay, KL penalties in VAEs/RLHF) are added to the task loss, not substitutes for it.

Gradients, Stability & Practical Pairings

  • Activation-loss pairing matters: softmax+CE and sigmoid+BCE both give the clean gradient (yhat - y) at the logits, which is well-scaled and stable.
  • Vanishing gradients: from sigmoid/tanh saturation and deep stacks; mitigate with ReLU-family, residual connections, normalization, careful init.
  • Exploding gradients: common in deep/recurrent nets; mitigate with gradient clipping, normalization, and proper init.
  • Dying ReLU: large learning rate or bad init pushes pre-activations permanently negative; use Leaky ReLU/GELU or lower the LR.
  • Always feed raw logits to framework losses (CrossEntropyLoss, BCEWithLogitsLoss); never apply softmax/sigmoid first, to avoid double-activation and instability.
  • Use log1p/logsumexp and epsilon-guarded logs to prevent log(0)/overflow; reduction mode (mean vs sum) changes effective gradient magnitude and LR scaling.
Read the full lesson