Deep learning

BatchNorm estimates mean/variance per mini-batch, so with very small batches (e.g., 2-4) these statistics are noisy and biased, degrading both training stability and the running-statistic estimates used at inference. Group Normalization, Layer Normalization, and Instance Normalization avoid this by normalizing across feature dimensions independently of batch size. GroupNorm in particular is the common drop-in replacement for detection/segmentation where memory limits batch size.

FlashAttention is IO-aware: it computes attention in tiles using online softmax so that the full N×N attention matrix is never materialized in slow HBM, drastically reducing memory reads/writes. This makes it memory-linear in sequence length (O(N) extra memory instead of O(N^2)) and faster despite recomputation, because the bottleneck is memory bandwidth, not FLOPs. It is exact attention, not an approximation.

Self-attention is permutation-equivariant, so without positional information the model cannot distinguish token order. Absolute sinusoidal encodings add a fixed position-dependent vector to embeddings, whereas RoPE rotates query/key vectors by an angle proportional to position, encoding relative position directly in the dot product. RoPE generalizes better to longer contexts (especially with interpolation/scaling) and is the standard in modern LLMs because relative offsets are preserved multiplicatively.

During autoregressive decoding, the KV cache memory and bandwidth scale with the number of key/value heads, which dominates inference cost for long contexts. GQA shares a smaller set of K/V heads across groups of query heads, shrinking the KV cache (and memory bandwidth) while retaining most of multi-head's expressiveness. It interpolates between full Multi-Head Attention and Multi-Query Attention (single K/V head).

Residual connections add the input directly to the output of a block (y = F(x) + x), creating an identity shortcut that lets gradients flow unimpeded during backprop, mitigating vanishing gradients. They reframe layers as learning residual corrections to identity rather than full transformations, which is easier to optimize. This is why ResNets and Transformers can stack hundreds of layers without degradation.

LoRA freezes the pretrained weights W and learns a low-rank update ΔW = (alpha/r)·BA (with rank r ≪ dimensions), so only A and B are trained and the rest stays fixed. This reduces trainable parameters and optimizer memory by orders of magnitude while approximating full fine-tuning quality, because weight updates during adaptation tend to be low-rank. At inference, BA can be merged into W, adding zero latency.

# Merge at inference: W_eff = W + (alpha/r) * B @ A

Adam adapts per-parameter learning rates using first and second moment estimates of gradients, which handles sparse gradients and ill-conditioned loss surfaces well, giving fast early convergence. However, adaptive methods can converge to sharper minima that sometimes generalize worse than SGD with momentum, and Adam's L2 regularization is coupled with the adaptive step. AdamW fixes the latter by decoupling weight decay from the gradient update.

In vanilla RNNs, gradients are repeatedly multiplied by the recurrent weight matrix through time, causing them to shrink (or explode) exponentially over long sequences, so long-range dependencies cannot be learned. LSTMs introduce a cell state with additive updates gated by input/forget/output gates, providing a near-constant error carousel that lets gradients flow over many timesteps. The forget gate learns when to retain or discard memory.

The forward process gradually adds Gaussian noise to data over T steps according to a fixed schedule, eventually turning the sample into near-pure noise. The network (a U-Net or, increasingly, a transformer/DiT) is trained to reverse this, most commonly by predicting the noise epsilon added at each step (equivalently the score, or a v-prediction target). Sampling then iteratively denoises from random noise back to a data sample.

CFG trains a single conditional model that randomly drops the condition during training, so at inference you can compute both conditional and unconditional noise predictions and extrapolate: eps = eps_uncond + s*(eps_cond - eps_uncond). The scale s controls the tradeoff between sample fidelity/prompt-adherence (high s) and diversity (low s); too high s causes oversaturation and reduced variety. It avoids needing a separate classifier.

LayerNorm normalizes across the feature dimension for each token independently, making it invariant to batch size and sequence length, which suits variable-length sequences and small/streaming batches. BatchNorm's dependence on batch statistics is problematic with the highly variable token statistics in NLP and during autoregressive inference. Pre-LN placement (normalizing before sublayers) further stabilizes deep Transformer training.

A ReLU unit outputs zero for all negative inputs and has zero gradient there, so if a neuron's weights push it permanently into the negative region it stops updating ('dies'). Leaky ReLU allows a small negative slope so gradients still flow, while GELU/ELU provide smooth nonzero gradients for negatives. GELU is standard in Transformers because its smoothness improves optimization.

In an MoE layer, a router selects only the top-k experts (e.g., 1-2 of many) per token, so only a small fraction of the total parameters are activated for any given token. This decouples total parameter count (capacity) from per-token FLOPs, enabling much larger models at constant inference compute. The challenge is load balancing so experts are utilized evenly, usually via an auxiliary balancing loss.

Temperature T scales logits before softmax (logits/T), controlling the entropy of the output distribution. T<1 sharpens the distribution toward high-probability tokens (more deterministic, less diverse), while T>1 flattens it (more random/creative). T→0 approaches greedy/argmax decoding.

probs = softmax(logits / temperature)

Top-k always keeps a fixed number of candidates regardless of the distribution's shape, which can include unlikely tokens when the model is confident or cut good tokens when it is uncertain. Top-p instead keeps the smallest set of tokens whose cumulative probability exceeds p, adapting the candidate count to the model's confidence at each step. This yields more coherent yet diverse generations.

Gradient (activation) checkpointing saves only a subset of activations during the forward pass and recomputes the rest during the backward pass, reducing peak memory at the cost of extra computation (roughly one additional forward pass). This trades compute for memory, enabling training of larger models or longer sequences on fixed hardware. It is widely used alongside FlashAttention and ZeRO.

Data parallelism replicates the full model on each device and splits the batch, synchronizing gradients via all-reduce; it scales throughput but each device must hold the whole model. Tensor parallelism shards individual layers' weight matrices across devices, splitting the matmul itself, so a single layer's computation is distributed—necessary when a model is too large for one device. They are often combined with pipeline parallelism in 3D parallelism.

ZeRO eliminates memory redundancy in data-parallel training by partitioning optimizer states, gradients, and parameters across devices instead of replicating them. Stage 1 shards optimizer states, Stage 2 also shards gradients, and Stage 3 additionally shards parameters, with each device reconstructing needed shards via communication (all-gather) on demand. This lets data parallelism train models far larger than a single device's memory.

Label smoothing replaces one-hot targets with a soft distribution (e.g., 0.9 for the true class, spreading 0.1 over the rest), preventing the model from becoming overconfident and producing extreme logits. This regularizes by encouraging smaller logit gaps, improving calibration and often accuracy. The downside is it can slightly degrade representations used for downstream retrieval/distillation due to tighter, less informative clusters.

Knowledge distillation trains a smaller student to match a larger teacher's output distribution (soft targets), often with a temperature to soften logits. Soft labels carry 'dark knowledge'—the relative probabilities across wrong classes encode similarity structure the teacher learned—providing a richer training signal than one-hot labels. This lets compact students approach teacher accuracy with far less compute.

The temperature scales the similarities (logits) in the InfoNCE softmax, controlling how much the loss penalizes hard negatives. A low temperature sharpens focus on the hardest negatives (pushing toward more uniform/spread embeddings) but can be unstable, while a high temperature treats negatives more uniformly. Tuning it is critical for the quality of learned representations.

Aleatoric uncertainty is inherent noise in the data (e.g., sensor noise, label ambiguity) that cannot be reduced by more data. Epistemic uncertainty is model uncertainty from limited data or knowledge, which shrinks as you collect more data and is captured via Bayesian methods, deep ensembles, or MC dropout. Distinguishing them matters for active learning and safe deployment.

Mixed precision stores activations/weights in 16-bit to halve memory and exploit faster tensor-core matmuls while keeping a master FP32 copy for stable updates. FP16 has a narrow exponent range, requiring loss scaling to avoid underflow/overflow; BF16 has the same exponent range as FP32 (but fewer mantissa bits), so it rarely needs loss scaling and is more numerically robust for large models.

Operations like quantization, rounding, or argmax have zero or undefined gradients, blocking backpropagation. The STE applies the discrete op in the forward pass but approximates its gradient as identity (or a clipped identity) in the backward pass, allowing gradients to flow through. It is foundational for quantization-aware training and VQ-VAE codebook learning.

# forward: y = round(x); backward: dy/dx = 1
y = x + (round(x) - x).detach()

Standard self-attention computes pairwise interactions, giving O(N^2 · d) time and O(N^2) memory in sequence length N, which dominates for long contexts. Approaches to reduce it include sparse attention (e.g., Longformer/BigBird sliding-window + global), linear attention/kernel methods (Performer), state-space models (Mamba), and IO-aware exact methods (FlashAttention reduces memory but not asymptotic FLOPs). Each trades exactness or expressiveness for efficiency.

Teacher forcing feeds the ground-truth previous token during training, but at inference the model conditions on its own (possibly erroneous) predictions, creating a train/test distribution mismatch. Errors can compound over the sequence because the model never learned to recover from its own mistakes. Mitigations include scheduled sampling, sequence-level training objectives, and RL-based fine-tuning.

During PPO fine-tuning, the policy is optimized to maximize the reward model's score, but unconstrained optimization leads to reward hacking and degenerate text. A per-token KL penalty between the current policy and the frozen SFT reference policy keeps outputs close to the original distribution, preventing the model from drifting into out-of-distribution gibberish that exploits the reward model. The KL coefficient balances reward maximization against staying coherent.

DPO reframes preference alignment as a single supervised classification-style loss over preferred vs. rejected pairs, implicitly optimizing the same KL-constrained reward objective without training a separate reward model or running online RL. This removes the instability and complexity of PPO rollouts and reward-model serving. It is simpler and more stable, though PPO can still outperform with high-quality online data and reward models.

With plain SGD, L2 regularization (adding λ‖w‖² to the loss) is equivalent to weight decay. But with Adam, L2 regularization's gradient gets scaled by the per-parameter adaptive learning rate, so heavily-updated weights get effectively less decay—breaking the equivalence. AdamW decouples weight decay by applying it directly to weights after the adaptive update, restoring proper regularization.

Early in training, Adam's second-moment estimates are unreliable and attention/LayerNorm dynamics are unstable, so a large initial learning rate can cause divergence or bad early updates. Warmup ramps the LR up (typically linearly) from near zero over the first few thousand steps, stabilizing these early updates before decaying. It is especially important for large batch sizes and deep post-LN Transformers.

A small MLP projection head maps encoder features to the space where the contrastive loss is applied. The head absorbs information specific to the contrastive task (e.g., invariances to augmentations) that would be harmful if baked into the encoder representation. Discarding it and using the pre-projection features yields better, more general downstream representations.

Convolutions apply shared local filters across spatial positions, giving translation equivariance and a locality prior that makes CNNs data-efficient on images. ViTs treat image patches as tokens with global self-attention and lack these built-in biases, so they need large datasets (or strong augmentation/distillation) to learn spatial structure, but scale better and capture long-range dependencies once enough data is available.

Catastrophic forgetting is when a network trained sequentially on new tasks overwrites weights important for previous tasks, sharply degrading old-task performance. Mitigations include regularization-based methods (e.g., Elastic Weight Consolidation penalizing changes to important weights), replay/rehearsal of past examples, and parameter-isolation methods (separate subnetworks/adapters per task). The core challenge is the stability-plasticity tradeoff.

The KL term regularizes the approximate posterior q(z|x) toward the prior p(z), enabling a smooth, sampleable latent space and acting as the 'rate' in a rate-distortion tradeoff. If the decoder is too powerful, the model can ignore the latent code and drive KL to zero—'posterior collapse'—producing uninformative latents. Fixes include KL annealing, free bits, and weakening the decoder.

Initialization sets the variance of activations and gradients at the start; if too large or small, signals explode or vanish through depth before training can correct them. Xavier (Glorot) init preserves variance assuming linear/tanh activations, while He init accounts for ReLU's halving of variance by scaling by 2/fan_in. Proper init keeps forward activations and backward gradients well-scaled, enabling stable training.

Speculative decoding uses a small, fast draft model to propose several future tokens, which the large target model then verifies in a single parallel forward pass; accepted tokens are kept and the first rejection triggers resampling from an adjusted distribution. Because the target verifies multiple tokens at once (memory-bandwidth-bound), it produces several tokens per large-model pass. A correct acceptance rule guarantees the output distribution is identical to sampling from the target model alone.

Quantization maps weights/activations to low-bit integers to shrink memory and speed up matmuls, but a few activation channels with very large magnitudes (outliers) dominate the dynamic range, causing large errors when naively quantized. Methods like LLM.int8() keep outlier channels in higher precision (mixed-precision decomposition), while SmoothQuant migrates the scale difficulty from activations to weights. Weight-only schemes (GPTQ, and activation-aware AWQ) compensate using calibration data.

Selective state-space models process sequences with a recurrence/convolution that scales linearly in sequence length and maintains a fixed-size hidden state, giving O(N) compute and constant memory per step at inference—unlike attention's O(N^2) and growing KV cache. Mamba adds input-dependent (selective) state transitions to retain content-based reasoning. The tradeoff is potentially weaker exact recall of arbitrary past tokens compared to full attention.

Modern deep networks are overparameterized and trained with cross-entropy to high confidence, so they often become overconfident—predicted probabilities exceed true accuracy—even with low training loss. Temperature scaling, a single scalar that divides logits and is fit on a validation set, is a simple, effective post-hoc calibration that doesn't change the predicted class. Calibration metrics like Expected Calibration Error (ECE) quantify the gap.

In pipeline parallelism, the model is split into stages across devices; the bubble is the idle time at the start/end when later stages wait for earlier stages to produce activations (and vice versa in backward). Splitting a batch into many micro-batches keeps all stages busy in a staggered schedule (e.g., GPipe/1F1B), shrinking the bubble fraction roughly to (stages-1)/(stages-1+micro-batches). More micro-batches reduce idle time at the cost of more activation memory.