Skip to content
elevatedevco

Cheat sheet · Deep Learning

Neural Network Basics cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

Cheat sheet

A dense reference on the core mechanics of feedforward neural networks: the neuron/layer model, forward and backward propagation, activations, losses, optimizers, initialization, and regularization — the foundations interviewers expect you to derive and reason about from scratch.

The Neuron, Layers & Forward Pass

  • A neuron computes a weighted sum plus bias, then a nonlinearity: a = f(w·x + b). Stacking neurons into layers gives the MLP (multilayer perceptron), the canonical fully-connected feedforward net.
  • Forward pass per layer l: z = Wx + b; a = f(z). For a single sample, W has shape (n_out, n_in), x is (n_in,), z and a are (n_out,). One matmul vectorizes the whole layer; batching stacks samples for a second matmul dimension.
  • Depth (more layers) buys hierarchical/compositional features; width (more units) buys capacity per layer. The Universal Approximation Theorem says one sufficiently wide hidden layer CAN approximate any continuous function on a compact set, but depth often reaches the same accuracy with exponentially fewer parameters.
  • Without nonlinear activations, stacked linear layers collapse to a single linear map (W2(W1x) = (W2W1)x) — nonlinearity is what makes a network more expressive than logistic/linear regression.
  • Bias terms shift the pre-activation; they are typically dropped only when immediately followed by a normalization layer (e.g., BatchNorm) whose learnable shift makes them redundant.
  • Parameter count of a dense layer = n_in * n_out + n_out (weights + biases). Memorize this; interviewers ask you to size models and estimate memory.

Activation Functions

  • ReLU: f(x)=max(0,x). Cheap, induces sparsity, no vanishing gradient for x>0; the default for hidden layers. Risk: 'dying ReLU' (neurons stuck outputting 0 with zero gradient).
  • Leaky ReLU / PReLU / ELU keep dead units alive via a small negative slope. GELU (x·Φ(x)) is the de-facto choice in Transformers; SiLU/Swish (x·σ(x)) is common in modern CNNs; gated variants like SwiGLU dominate 2024–2026 LLM feedforward blocks.
  • Sigmoid σ(x)=1/(1+e^-x) maps to (0,1) — use for binary/multilabel outputs or gates, NOT hidden layers (saturates, gradients vanish, output not zero-centered).
  • Tanh maps to (-1,1) and is zero-centered, but still saturates. Softmax normalizes a logit vector into a probability distribution for single-label multiclass output.
  • Choose activations for gradient flow: you want derivatives near 1 in the active region to avoid vanishing/exploding gradients across depth.
  • Output-layer activation is dictated by the task: linear/identity (regression), sigmoid (binary or multilabel), softmax (single-label multiclass).

Loss Functions

  • Regression: Mean Squared Error L=mean((y-ŷ)^2); MAE for robustness to outliers; Huber blends the two (quadratic near 0, linear in the tails).
  • Binary classification: binary cross-entropy L=-[y·log p + (1-y)·log(1-p)] with a sigmoid output.
  • Multiclass: categorical cross-entropy L=-Σ y_i log p_i with softmax. With integer (not one-hot) labels use the sparse variant.
  • For numerical stability, frameworks fuse softmax+cross-entropy (logits-based loss using the log-sum-exp trick) — pass raw logits, not probabilities.
  • Cross-entropy equals the negative log-likelihood under the model's predicted distribution; minimizing it is maximum-likelihood estimation.
  • Use class weights, resampling, or focal loss for class imbalance; the loss must be paired with the matching output activation.

Backpropagation & Gradient Descent

  • Backprop = reverse-mode automatic differentiation: apply the chain rule from the loss backward through the compute graph, reusing intermediate activations cached during the forward pass.
  • Per-layer gradients: dL/dz = dL/da ⊙ f'(z); dL/dW = dL/dz · a_prevᵀ; dL/db = dL/dz (summed over the batch); propagate dL/da_prev = Wᵀ · dL/dz.
  • Update rule (SGD): θ ← θ − η · ∇θ L, where η is the learning rate — the single most important hyperparameter.
  • Batch GD (whole dataset; stable but slow), SGD (one sample; noisy), Mini-batch (typical, e.g. 32–512) trades gradient variance for hardware throughput.
  • Vanishing/exploding gradients come from repeated multiplication of small/large Jacobians across depth; mitigate with ReLU-family activations, good init, residual connections, normalization, and gradient clipping.
  • Backprop costs about the same as one forward pass; memory scales with stored activations — gradient checkpointing trades extra compute for less memory.

Optimizers & Learning-Rate Schedules

  • Momentum accumulates a velocity v = βv + ∇L; θ ← θ − ηv. It dampens oscillation and speeds convergence along ravines.
  • Adam combines momentum (1st moment) with RMSProp-style per-parameter scaling (2nd moment) plus bias correction — a robust default. AdamW decouples weight decay from the gradient step and is standard for Transformers.
  • Adaptive methods converge fast but can generalize slightly worse than well-tuned SGD+momentum on some vision benchmarks.
  • Schedules matter: linear/cosine warmup (ramp LR up early to stabilize), then cosine or step decay. Warmup is near-mandatory for large Transformers.
  • Typical knobs: Adam β1=0.9, β2=0.999, ε=1e-8; LR ~1e-3 (Adam) or ~0.1 (SGD with decay). Tune LR first.
  • Gradient clipping (by global norm) prevents exploding updates, especially in RNNs and deep/large-batch training.

Initialization, Normalization & Regularization

  • Init: never all-zeros or all-equal (symmetry never breaks — every neuron learns the same thing). Use Xavier/Glorot for tanh/sigmoid, He/Kaiming for ReLU, scaling variance by fan-in/fan-out to keep activation and gradient variance stable across depth.
  • BatchNorm normalizes per-mini-batch during training and uses running statistics at inference; LayerNorm normalizes per-sample over features and is the norm of choice in Transformers (RMSNorm is the lighter variant dominant in 2024–2026 LLMs).
  • Normalization stabilizes and accelerates training and permits higher learning rates; mind the differing train vs eval behavior (BatchNorm especially).
  • Dropout randomly zeros activations with probability p during training (with inverted scaling so expectations match) to curb co-adaptation; it is disabled at eval.
  • L2 weight decay penalizes large weights (smoother functions); L1 induces sparsity. Early stopping halts training when validation loss starts rising.
  • Residual/skip connections (y = x + F(x)) are essential for very deep nets — they give gradients a clean identity path and underpin ResNets and Transformers.
Read the full lesson