Diffusion models

The forward process is a fixed (non-learned) Markov chain that gradually adds Gaussian noise to data over T steps according to a predefined variance schedule: q(x_t|x_{t-1}) = N(x_t; sqrt(1-beta_t) x_{t-1}, beta_t I). It has no trainable parameters; only the reverse process is learned. A key property is that x_t can be sampled directly from x_0 in closed form using the cumulative product of (1-beta).

Using alpha_bar_t = product_{s=1}^t (1-beta_s), we have x_t = sqrt(alpha_bar_t) x_0 + sqrt(1-alpha_bar_t) epsilon with epsilon ~ N(0,I). This lets us sample any timestep t in O(1) without iterating the chain, enabling efficient stochastic-gradient training on a randomly chosen t each step.

# alpha_bar = torch.cumprod(1 - beta, dim=0)
x_t = alpha_bar[t].sqrt() * x0 + (1 - alpha_bar[t]).sqrt() * eps

It trains an epsilon-prediction network to predict the noise added to x_0, minimizing E[||epsilon - epsilon_theta(x_t, t)||^2]. This is a reweighted, simplified version of the variational ELBO that drops the time-dependent weighting terms (effectively weighting all timesteps equally) and empirically improves sample quality.

Predicting noise epsilon_theta(x_t,t) is equivalent (up to a known scaling factor) to estimating the score, the gradient of the log density of the noised data: score = grad_x log q(x_t) = -epsilon/sqrt(1-alpha_bar_t). Thus denoising score matching and DDPM noise prediction are two views of learning the same quantity.

DDIM defines a non-Markovian reverse process that shares the same training objective but admits a deterministic (eta=0) sampler. This decouples the number of sampling steps from the training T, enabling far fewer steps (e.g., 20-50) with consistent outputs and a meaningful, near-invertible latent-to-image mapping. With eta=1 DDIM recovers the stochastic DDPM sampler.

The score is shifted by the gradient of a separately trained noise-robust classifier: grad_x log p(x_t|y) = grad_x log p(x_t) + s * grad_x log p_phi(y|x_t), where s is the guidance scale. In practice this is applied as eps_hat = eps_theta - s*sqrt(1-alpha_bar_t)*grad_x log p_phi(y|x_t). Larger s trades diversity for class fidelity, and it requires an auxiliary classifier trained on noisy images.

CFG trains a single conditional model that randomly drops the condition during training (e.g., 10-20% of the time) so it learns both conditional and unconditional scores. At sampling it extrapolates: eps = eps_uncond + w*(eps_cond - eps_uncond). Its advantage is that no separate classifier on noisy data is needed, and it gives strong control over the fidelity/diversity trade-off via w.

eps = eps_uncond + w * (eps_cond - eps_uncond)

Higher w increases adherence to the conditioning (sharper, more prompt-aligned samples) at the cost of reduced diversity and, at very high values, oversaturation/artifacts and mode collapse. There is a sweet spot (commonly ~5-8 for text-to-image) balancing fidelity and realism.

LDMs run the diffusion process in the compressed latent space of a pretrained autoencoder (e.g., a VAE) rather than pixel space. This drastically reduces compute and memory while a perceptual+adversarial autoencoder preserves detail, with conditioning injected via cross-attention into the U-Net.

Text tokens are encoded (e.g., by CLIP or T5) and injected via cross-attention layers where the U-Net feature maps form queries and the text embeddings form keys/values. This lets spatial features attend to relevant words at multiple resolutions.

The network must behave differently at different noise levels (early reverse steps refine fine detail at low noise, late reverse steps shape coarse structure at high noise). The timestep is encoded (sinusoidal then MLP) and injected (often via FiLM-style scale/shift in group-norm) so each block conditions on the current noise scale.

Its encoder-decoder with skip connections preserves high-frequency spatial detail (via skips) while the bottleneck captures global context, and multi-resolution processing matches the need to denoise both coarse structure and fine texture. Attention at lower resolutions adds long-range dependencies.

The schedule sets how quickly signal is destroyed across timesteps. A cosine schedule (Nichol & Dhariwal) adds noise more gradually near the end, avoiding the rapid information loss of a linear schedule at high t, which improves likelihood and sample quality, especially at lower resolutions.

The forward process is a stochastic differential equation that gradually perturbs data into noise (e.g., VP-SDE corresponds to DDPM, VE-SDE to NCSN/SMLD). The reverse-time SDE, driven by the learned score grad_x log p_t(x), generates data from noise; a corresponding deterministic probability-flow ODE shares the same marginals.

It is a deterministic ODE whose trajectories have the same marginal distributions p_t(x) as the reverse-time SDE. Because it is deterministic, it enables fast deterministic sampling, exact likelihood computation via the instantaneous change-of-variables, and a unique latent encoding of each sample.

Consistency models enable one-step (or few-step) generation by learning a function f that maps any point on a probability-flow ODE trajectory directly to the trajectory's origin (the clean data). The self-consistency property requires f(x_t,t) to be the same for all t on the same trajectory, allowing single-step sampling while still supporting optional multi-step refinement.

Flow matching trains a network to regress a target velocity field of a continuous-time ODE that transports a simple prior to data, often along straight-line (rectified) interpolations x_t=(1-t)x_0+t*noise. It generalizes diffusion (diffusion's probability-flow ODE is one instance) and straight paths enable very few-step sampling.

Instead of predicting epsilon or x_0, the network predicts v = sqrt(alpha_bar_t)*epsilon - sqrt(1-alpha_bar_t)*x_0. v-prediction gives stable, well-conditioned targets across all noise levels (including very high noise / SNR->0), improving training stability and enabling effective progressive distillation and zero-terminal-SNR training.

Progressive distillation trains a student model to take one step that matches two steps of a teacher's deterministic (DDIM) sampler, halving the step count each round. Repeating this distills a many-step sampler down to very few steps (e.g., 4-8) with minimal quality loss.

Many schedules leave a small residual signal at t=T (terminal SNR > 0), so the model is trained on slightly signal-bearing 'pure noise' but sampled from true N(0,I), causing a train/test mismatch that limits brightness/contrast range. Rescaling the schedule to enforce zero terminal SNR (typically with v-prediction) fixes this, allowing very dark or very bright images.

mu_theta(x_t,t) = (1/sqrt(1-beta_t)) * (x_t - (beta_t/sqrt(1-alpha_bar_t)) * epsilon_theta(x_t,t)). The variance is usually fixed to beta_t or the posterior variance (or optionally learned). One then samples x_{t-1} = mu_theta + sigma_t * z.

Frechet Inception Distance (FID) is standard: it computes the Frechet distance between Gaussian fits of Inception-v3 features for generated vs real images, capturing both fidelity and diversity. Lower is better; it is sensitive to sample count and the feature extractor.

Diffusion training is a stable regression (denoising) objective with no adversarial min-max game, avoiding mode collapse and instability. Because it optimizes a likelihood-based bound, it tends to cover modes well, at the cost of slower iterative sampling compared to a GAN's single forward pass.

SDEdit noises an input/guide image to an intermediate timestep t0 then runs the reverse process to synthesize a realistic image. A higher strength (larger t0) adds more noise, giving more creative freedom but less fidelity to the input; lower strength preserves structure but allows fewer changes.

At each reverse step, the known region is replaced by a correctly-noised version of the original image (forward-diffused to that timestep) while the model fills the masked region; the two are blended via the mask. This keeps the unmasked area consistent while letting the model harmonize the masked content.

ControlNet clones the encoder (and middle block) of a frozen pretrained diffusion U-Net into a trainable copy, connects it back with zero-initialized convolutions, and trains it on a conditioning signal (edges, depth, pose). Zero-init means initial outputs are unchanged, enabling stable fine-tuning that adds spatial control without degrading the base model.

A DiT replaces the convolutional U-Net with a Vision-Transformer backbone operating on latent patches, using adaptive layer norm (adaLN-zero) to inject timestep and class conditioning. DiTs scale better with compute/parameters and underpin modern large models like SD3 and Sora-style systems.

They factorize attention/convolutions into spatial and temporal layers (or use 3D/space-time attention) over a sequence of latent frames, often inflating a pretrained image model and adding temporal layers. Temporal attention plus shared noise/conditioning across frames enforces consistency, and many models also compress along the time axis.

Discrete tokens have no meaningful Gaussian noise or continuous interpolation, so standard Gaussian score matching does not apply. Solutions include discrete/multinomial diffusion (D3PM) using categorical transition kernels (e.g., absorbing/uniform), continuous diffusion in embedding space (Diffusion-LM), or masked/absorbing-state formulations that gradually corrupt tokens.

They are higher-order ODE solvers tailored to the semi-linear structure of the probability-flow ODE, exploiting an exponential-integrator / analytic treatment of the linear term so each step is more accurate. This yields high-quality samples in ~10-20 steps, fewer than first-order DDIM.

The negative ELBO decomposes into a reconstruction term (L0 = -log p(x_0|x_1)), a sum of KL divergences L_{t-1}=KL(q(x_{t-1}|x_t,x_0) || p_theta(x_{t-1}|x_t)) for t=2..T matching the learned reverse step to the tractable forward posterior, and a prior term L_T=KL(q(x_T|x_0)||N(0,I)) that is ~0. Each KL term has a closed-form Gaussian expression.

VP (variance-preserving) SDE keeps the variance of the perturbed data bounded (~1) by scaling down the signal while adding noise; it corresponds to DDPM. VE (variance-exploding) SDE keeps the signal fixed and lets the noise variance grow large; it corresponds to score-matching / NCSN. They differ in how signal and noise magnitudes evolve.

At t=0 we want the clean data estimate (the mean), not a noisy sample, so the stochastic term sigma_t*z is set to zero on the last step (z=0). Adding noise there would inject unnecessary variance into the final output; for t>1 the noise term maintains the correct reverse-chain stochasticity.

z = torch.randn_like(x) if t > 1 else torch.zeros_like(x)
x = mu + sigma_t * z

Each guided step needs both eps_cond and eps_uncond, so implementations concatenate the conditioned and null (empty/dropped) embeddings into one batch and do a single forward pass, reducing kernel-launch/overhead versus two separate calls. The two outputs are then combined with the guidance scale.

out = model(torch.cat([x, x]), t, torch.cat([null_emb, cond_emb]))
eps_u, eps_c = out.chunk(2)
eps = eps_u + w * (eps_c - eps_u)