Skip to content
elevatedevco

Cheat sheet · Deep Learning

Dropout & Batch Normalization cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

Cheat sheet

Dropout is a stochastic regularizer that randomly zeroes activations to break co-adaptation of features, while Batch Normalization stabilizes and accelerates training by normalizing layer inputs per mini-batch. Both have distinct train vs. eval behaviors that are a frequent source of bugs and a near-guaranteed interview topic.

Dropout: Mechanism & Intuition

  • During training, each unit is KEPT with probability p (dropped with 1-p), forcing the network not to rely on any single feature and reducing co-adaptation.
  • Can be viewed as training an exponential ensemble of weight-sharing sub-networks; inference approximates averaging over that ensemble.
  • Inverted dropout (the standard implementation) scales kept activations by 1/p at TRAIN time, so the expected activation is unchanged and inference needs no rescaling.
  • At evaluation/inference, dropout is OFF (identity) and the full network with all units active is used.
  • Typical rates: ~0.5 for fully-connected hidden layers, 0.1-0.2 for inputs; higher dropout means stronger regularization but slower convergence. Note: PyTorch nn.Dropout(p) uses p as the DROP probability.

Batch Normalization: Mechanism & Formula

  • Normalizes each feature/channel across the mini-batch to zero mean and unit variance, then applies a learnable affine transform (gamma, beta) so the layer can recover any needed scale/shift.
  • Originally motivated as reducing 'internal covariate shift'; the modern consensus is that its main benefit is smoothing the loss landscape, enabling higher learning rates and faster, more stable convergence.
  • Maintains running (exponential moving average) estimates of mean and variance during training; these fixed statistics are used at inference.
  • Adds a mild regularization effect because each example's normalization depends on the other examples in the batch (batch noise).
  • gamma and beta are learned per-feature; gamma can in principle undo the normalization, so BN does not reduce representational power.

Train vs. Eval Mode (Critical)

  • Both layers behave differently in train vs. eval; forgetting to switch modes is the #1 practical bug.
  • Dropout: active in train (mask + 1/p scaling), identity in eval.
  • BatchNorm: uses batch statistics in train (and updates running stats), uses stored running stats in eval.
  • In PyTorch call model.train() / model.eval(); in TF/Keras the `training` flag flows through fit/predict automatically.
  • For validation/inference always set eval mode AND disable gradient tracking (torch.no_grad() / inference_mode()) for correctness and speed.

Normalization Variants & When to Use

  • LayerNorm: normalizes across features within a single sample; batch-size independent; the default in Transformers/LLMs and for RNNs.
  • GroupNorm: splits channels into groups; strong for small batches and detection/segmentation where BN statistics are unreliable.
  • InstanceNorm: per-sample, per-channel; common in style transfer and generative imaging.
  • RMSNorm: LayerNorm without mean-centering (scales by RMS only, optional gain); cheaper and standard in modern LLMs (LLaMA, Gemma, Qwen).
  • BatchNorm shines in CNNs with reasonably large batches; it struggles with very small batches, sequence models, and online/streaming settings.

Interactions, Placement & Practical Tips

  • BN reduces the need for dropout; combining them in CNNs often hurts due to a train/inference variance mismatch ('disharmony', Li et al. 2019).
  • If using both, apply dropout AFTER BN, typically near the final dense layers, not between conv-BN blocks.
  • Disable the conv/linear bias when followed by BN (bias=False) — beta absorbs the shift, so the bias is redundant.
  • Common order: Conv/Linear -> BN -> activation (ReLU). BN-before-activation is the conventional default, though some variants differ.
  • BN couples examples in a batch, so it interacts with batch size, gradient accumulation, and distributed training (use SyncBN across GPUs).
  • Pre-norm (LayerNorm before the sublayer) is standard in modern deep Transformers for stable training; post-norm needs careful warmup.

Regularization Effect & Hyperparameters

  • Dropout strength is tuned via the rate; treat it as a regularization knob alongside weight decay and data augmentation.
  • BN's regularization comes from batch noise: larger batches give cleaner stats but weaker regularization.
  • BN typically permits a higher learning rate and shortens warmup, speeding convergence.
  • eps (e.g., 1e-5) prevents divide-by-zero; PyTorch BN momentum (default 0.1) is the weight on the NEW batch stats in the EMA update.
  • DropConnect, Spatial/2D Dropout (drops whole feature maps/channels), and Stochastic Depth (drops residual blocks) are task-specific dropout variants worth knowing.
Read the full lesson