Skip to content
elevatedevco

Cheat sheet · Deep Learning

RNN, LSTM & GRU cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

Cheat sheet

Recurrent networks process sequences by maintaining a hidden state across time steps; LSTM and GRU add gating plus an additive memory path to fight vanishing gradients and capture long-range dependencies. Though largely superseded by Transformers for large-scale NLP, they remain strong, efficient choices for low-latency, streaming, and small-data time-series problems.

Vanilla RNN: Mechanics & Limits

  • Recurrence: the hidden state is a function of the current input and the previous hidden state, sharing the SAME weight matrices across all time steps (parameter sharing enables variable-length sequences).
  • Trained with Backpropagation Through Time (BPTT): unroll the network over the sequence, then backprop; the gradient w.r.t. early steps is a product of many Jacobians.
  • Vanishing/exploding gradients: repeated multiplication by the recurrent weight makes gradients shrink (spectral radius < 1) or blow up (> 1) over long horizons, so plain RNNs rarely learn dependencies beyond ~10-20 steps.
  • tanh is the usual activation (zero-centered, bounded); ReLU in RNNs tends to explode without careful init (e.g., IRNN: identity-initialized recurrent weights + zero bias).
  • Truncated BPTT bounds memory/compute by backpropagating only k steps; it trades long-range gradient signal for tractability (the forward pass still carries state across the full sequence).

LSTM: Gated Long-Term Memory

  • Adds a cell state C_t (the 'conveyor belt') alongside hidden state h_t; the additive cell update gives a near-constant gradient path (the Constant Error Carousel), mitigating vanishing gradients.
  • Three gates (sigmoid, 0..1): forget f_t (what to drop from the cell), input i_t (what to write), output o_t (what to expose as h_t); the candidate g_t uses tanh.
  • Cell update is ADDITIVE: C_t = f_t * C_{t-1} + i_t * g_t — this additive path, not the gates alone, is why gradients survive over long horizons.
  • ~4x the parameters of a vanilla RNN cell (4 weight sets). Initialize the forget-gate bias to +1 (or higher) so the cell remembers by default early in training.
  • Variants: peephole connections let gates also see C_{t-1}; most frameworks default to the no-peephole form. Bidirectional LSTM reads the sequence both directions (offline only — needs the full sequence).

GRU: Streamlined Gating

  • Merges cell and hidden state into a single h_t and uses only two gates: update z_t and reset r_t — 3 weight sets, fewer params than LSTM.
  • Update gate z_t interpolates between the previous state and the new candidate, acting like a combined forget+input gate (leaky integration).
  • Reset gate r_t controls how much past state feeds the candidate, letting the unit drop irrelevant history.
  • No separate output gate and no exposed cell state, so the full state is always passed forward (less control, often comparable accuracy).
  • Convention note: the original (Cho) form below uses h_t = (1 - z_t)*h_{t-1} + z_t*n_t; PyTorch's nn.GRU swaps it to h_t = (1 - z_t)*n_t + z_t*h_{t-1} (z then gates the OLD state). Same family, different gate orientation.
  • Rule of thumb: GRU trains faster and does well on smaller datasets; LSTM can edge it out on very long or complex dependencies. Empirically the two are close — benchmark both.

Training Best Practices

  • Gradient clipping (by global norm, e.g. 1.0-5.0) is essentially mandatory to prevent exploding gradients.
  • Regularize with recurrent/variational dropout (same mask reused across time steps), zoneout, and weight decay; naive per-step dropout on recurrent connections hurts memory.
  • Pack padded sequences (e.g. pack_padded_sequence) so the RNN skips PAD tokens; never let padding pollute the loss — mask it out.
  • Use layer normalization for RNNs (batch norm is awkward across time). Orthogonal init for recurrent weights helps gradient flow.
  • Mind the data layout (batch_first vs seq_first) and use cuDNN-fused kernels for speed. Use TBPTT for very long sequences and bucket by length to reduce padding.

RNN/LSTM/GRU vs Transformers (2026 view)

  • RNNs are O(T) sequential — they cannot parallelize across time during training; Transformers are O(T^2) attention but fully parallel over positions, which is why they dominate large-scale NLP.
  • RNNs excel where Transformers struggle: low-memory/streaming inference (constant state, no growing KV cache), strict causal online prediction, and small datasets.
  • Inference memory: an RNN's recurrent state is O(1) in sequence length; a Transformer's KV cache grows O(T) — a real cost for long contexts and edge devices.
  • Modern resurgence: state-space models (S4, Mamba) and linear-attention/RNN hybrids like RWKV revive recurrence — parallel training plus linear-time, constant-memory inference.
  • Practical defaults: time-series forecasting, sensor/IoT, embedded/mobile, and low-latency tasks often still favor GRU/LSTM; reach for Transformers (or SSMs) when you have scale, long context, and parallel compute.

Architecture & Application Patterns

  • Sequence I/O modes: one-to-many (captioning), many-to-one (sentiment/classification — use last hidden state or pooled states), many-to-many aligned (tagging/POS), and seq2seq encoder-decoder (translation).
  • Encoder-decoder with attention: attention over encoder states fixed the information bottleneck of a single fixed context vector — the direct precursor to the Transformer.
  • Stacked/deep RNNs: feed h from layer L as input to layer L+1; add residual connections to ease training of deep stacks.
  • Bidirectional only for offline tasks (whole sequence available); use unidirectional for real-time/causal generation.
  • For classification, mean/max-pooling over all time steps often beats using only the final state, especially with a BiLSTM.
Read the full lesson