Skip to content
elevatedevco

Cheat sheet · Deep Learning

Optimizers (SGD, Adam) cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

Cheat sheet

Optimizers update model weights from gradients to minimize a loss function. SGD (with momentum) and Adam/AdamW are the workhorses of deep learning; knowing their update rules, hyperparameters, memory costs, and generalization trade-offs is essential for training models and for interviews.

Core Concept: Gradient Descent

  • All first-order optimizers minimize loss L(theta) by stepping parameters opposite the gradient: theta <- theta - eta * grad L(theta).
  • Batch GD uses the full dataset per step (stable but slow and memory-heavy); pure SGD uses one sample (noisy); mini-batch (typically 32-512) is the practical default, trading gradient variance for hardware/throughput efficiency.
  • The learning rate eta is the single most important hyperparameter: too high diverges or oscillates, too low trains slowly or stalls on plateaus.
  • Gradient noise from mini-batching acts as implicit regularization and helps escape saddle points and sharp regions of the loss surface.
  • Optimizers differ mainly in how they transform the raw gradient into the actual update: momentum (history), adaptive per-parameter scaling, or both.

SGD and Momentum

  • Vanilla SGD: theta <- theta - eta * g_t, where g_t is the mini-batch gradient. Simple, low memory, and well-understood generalization behavior.
  • Momentum keeps an exponentially-decayed velocity v to dampen oscillation and accelerate along consistent directions: v <- mu*v + g; theta <- theta - eta*v (mu ~= 0.9). This is the PyTorch convention; some texts write v <- mu*v + (1-mu)*g.
  • Nesterov momentum (NAG) evaluates the gradient at the look-ahead position (theta - eta*mu*v), giving a more responsive correction and often faster convergence.
  • SGD+momentum frequently generalizes better than adaptive methods on vision tasks (CNNs/ResNets) and remains standard for ImageNet-style training.
  • Almost always paired with an LR schedule (step decay, cosine annealing, or warmup) and weight decay for best results.
  • Drawback: needs careful LR tuning and is sensitive to feature scaling, since it applies one global learning rate to all parameters.

Adam: Adaptive Moment Estimation

  • Combines momentum (1st moment m) with per-parameter adaptive learning rates (2nd moment v, as in RMSProp).
  • Updates exponential moving averages: m <- b1*m + (1-b1)*g, and v <- b2*v + (1-b2)*g^2 (elementwise square).
  • Bias correction counters the zero-initialization of m and v early in training: m_hat = m/(1-b1^t), v_hat = v/(1-b2^t), where t is the step count.
  • Final step: theta <- theta - eta * m_hat / (sqrt(v_hat) + eps). Defaults: b1=0.9, b2=0.999, eps=1e-8, eta~=1e-3.
  • Robust to sparse gradients and poorly-scaled features; fast initial convergence and relatively forgiving of LR choice. The default for Transformers/NLP/LLMs.
  • AdamW decouples weight decay from the gradient-based update (it shrinks theta directly rather than adding L2 to the loss), fixing Adam's broken regularization, and is now standard for training large models.

SGD vs Adam: Choosing One

  • Adam/AdamW: faster convergence, less tuning, strong for sparse/noisy gradients, Transformers, RNNs, GANs, and rapid prototyping.
  • SGD+momentum: often better final generalization on CNNs/vision; tends to find flatter minima associated with lower test error.
  • Memory: Adam stores two extra states (m, v) per parameter, roughly 2x optimizer-state memory vs plain SGD, a real cost at large scale.
  • Common recipe: train with Adam/AdamW for speed, or switch to SGD with a strong cosine/step schedule when squeezing out final accuracy.
  • Learning-rate scales differ: Adam typically ~1e-3, SGD typically ~1e-1 to 1e-2. Never reuse the same LR across optimizers.
  • 2026 context: AdamW (with warmup + cosine decay) dominates LLM training; memory-efficient variants like 8-bit Adam, Adafactor, Lion, and Sophia cut optimizer-state overhead at scale.

Hyperparameters and Schedules

  • Learning rate: tune first; use an LR-range test or warmup to find a stable maximum, then decay.
  • Warmup (linearly ramp LR over the first few hundred/thousand steps) stabilizes early Adam training, critical for Transformers where early-step variance is high.
  • Cosine annealing and one-cycle policies are strong general-purpose schedules; step decay is simple and effective for vision.
  • Weight decay: use AdamW (decoupled), not L2-in-loss, for adaptive optimizers. Typical 0.01-0.1 for Transformers, ~1e-4 for SGD vision.
  • b1 controls momentum smoothing; b2 controls variance smoothing. Lowering b2 (e.g., 0.95) can improve very-large-batch LLM stability.
  • Gradient clipping (by global norm, e.g., max_norm=1.0) prevents exploding-gradient blowups, especially in RNNs and Transformers.

Related Optimizers (Context)

  • AdaGrad: accumulates squared gradients without decay, so per-parameter LR shrinks monotonically; good for sparse data but stalls on long runs.
  • RMSProp: fixes AdaGrad by using an EMA of squared gradients; effectively Adam without the momentum (1st-moment) term.
  • Adadelta: removes the need for a global LR by using a ratio of running averages of updates and gradients.
  • Lion (2023): sign-based update with only a momentum buffer, roughly half Adam's optimizer memory and competitive on large models.
  • Adafactor / 8-bit Adam: shrink optimizer-state memory (factored second moment / quantized states) for training huge models on limited hardware.
  • Second-order and approximate-curvature methods (Shampoo, Sophia, K-FAC) use curvature for faster convergence at extra compute/complexity.
Read the full lesson