Diffusion Models

Diffusion models learn to reverse a gradual noising process. The forward process slowly adds Gaussian noise to data over T steps until it becomes pure noise. The reverse process trains a neural network to denoise — iteratively removing noise to recover the original data. At inference, start from random Gaussian noise and apply the learned denoising steps T times to generate a new sample. Unlike GANs (adversarial) or VAEs (encoder-decoder), diffusion models are trained with a simple denoising objective and produce diverse, high-quality samples, becoming the dominant paradigm for image and audio generation.

The forward process adds Gaussian noise at each step: q(x_t | x_{t-1}) = N(x_t; √(1−β_t) x_{t-1}, β_t I), where β_t is the noise schedule. After T steps (T≈1000), x_T ≈ N(0,I). A key property: x_t can be sampled directly from x_0 without iterating: x_t = √ᾱ_t x_0 + √(1−ᾱ_t) ε, where ᾱ_t = Π_{s=1}^t (1−β_s) and ε ~ N(0,I). This closed form enables efficient training — sample a random t, add noise, train to denoise, without simulating all T steps.

import torch
def forward_diffuse(x0, t, sqrt_alpha_bar, sqrt_one_minus_alpha_bar):
    eps = torch.randn_like(x0)
    return sqrt_alpha_bar[t] * x0 + sqrt_one_minus_alpha_bar[t] * eps, eps

DDPM (Ho et al. 2020) trains a neural network ε_θ(x_t, t) to predict the noise ε that was added: L = E[||ε − ε_θ(x_t, t)||²]. At each training step: sample x_0 from data, sample t uniformly, sample ε ~ N(0,I), compute x_t = √ᾱ_t x_0 + √(1−ᾱ_t) ε, train ε_θ to predict ε. This simple MSE objective on noise prediction is equivalent (up to reweighting) to the variational lower bound on log p(x_0). Ho et al. found noise prediction outperforms predicting x_0 directly or the posterior mean x_{t-1}.

# DDPM training step
eps = torch.randn_like(x0)
t = torch.randint(0, T, (batch,))
x_t = sqrt_abar[t] * x0 + sqrt_1m_abar[t] * eps
loss = F.mse_loss(model(x_t, t), eps)

The reverse process models p_θ(x_{t-1}|x_t) = N(x_{t-1}; μ_θ(x_t,t), σ²_t I). Given the predicted noise ε_θ(x_t,t), the predicted mean is: μ_θ(x_t,t) = (1/√α_t)(x_t − (β_t/√(1−ᾱ_t)) ε_θ(x_t,t)). Sampling: x_{t-1} = μ_θ(x_t,t) + σ_t z, z ~ N(0,I). Starting from x_T ~ N(0,I), apply T reverse steps to obtain x_0. The reverse process is a Markov chain with learned transitions. With T=1000 steps, generation is slow (seconds per image) — motivating faster sampling methods like DDIM.

# DDPM reverse step
eps_pred = model(x_t, t)
mu = (x_t - beta[t]/sqrt_one_minus_abar[t] * eps_pred) / sqrt_alpha[t]
x_prev = mu + sigma[t] * torch.randn_like(x_t)

The noise schedule β_1,...,β_T controls how fast noise is added. DDPM uses a linear schedule: β_t linearly increases from β_1=0.0001 to β_T=0.02. Linear schedules add too much noise early, wasting steps. The cosine schedule (Nichol & Dhariwal 2021): ᾱ_t = cos²(πt/(2T+π·0.008)) — adds noise more slowly early (preserves more signal at small t) and has flat ends. Cosine schedule gives better FID scores, especially for low-resolution images where a large fraction of the image becomes pure noise too quickly with linear scheduling.

import numpy as np
def cosine_schedule(T, s=0.008):
    t = np.linspace(0, T, T+1)
    f = np.cos(((t/T + s)/(1+s)) * np.pi/2)**2
    alpha_bar = f / f[0]
    return np.clip(np.diff(alpha_bar) / alpha_bar[:-1], 0, 0.999)

DDIM (Song et al. 2021) enables faster sampling than DDPM by using a non-Markovian sampling process. DDIM defines a deterministic (or partially stochastic) reverse process with exactly the same training as DDPM. With 1000 training steps, DDIM samples using only 50–100 steps while maintaining quality. The key formula: x_{t-1} = √ᾱ_{t-1}·(x_t − √(1−ᾱ_t) ε_θ) / √ᾱ_t + √(1−ᾱ_{t-1}−σ_t²) ε_θ + σ_t z. With σ_t=0, DDIM is deterministic: same noise input → same output (useful for interpolation and editing).

The denoising network ε_θ(x_t, t) in DDPM uses a U-Net: an encoder that downsamples (conv + downsample blocks) and a decoder that upsamples (transposed conv/upsample + conv blocks) with skip connections from encoder to decoder at matching resolutions. Skip connections preserve fine-grained details lost in downsampling. The time step t is embedded via sinusoidal encoding, projected to a vector, and injected via AdaGN (adaptive group normalization) or added to residual blocks. Self-attention is added at lower-resolution feature maps for global consistency. U-Net is the dominant architecture for pixel-space diffusion (DDPM, Stable Diffusion's internal denoiser).

Score matching learns the score function s_θ(x) ≈ ∇_x log p(x) — the gradient of the log-density. Sampling via Langevin dynamics: x_{t+1} = x_t + δ s_θ(x_t) + √(2δ) z. Song & Ermon (2019) showed that denoising score matching (train s_θ to match ∇_x log q(x|x̃) for noisy data) is tractable. DDPM's noise prediction is equivalent: ε_θ(x_t,t) = −√(1−ᾱ_t) s_θ(x_t,t). So diffusion models are score-based generative models in disguise — they learn the score at multiple noise levels, enabling high-quality sampling via annealed Langevin dynamics.

Latent Diffusion Models (Rombach et al. 2022) apply diffusion in a compressed latent space rather than pixel space. A pre-trained VQ-VAE or KL-VAE encodes images to 64×64×4 latents (vs. 512×512×3 pixels — 48× smaller). Diffusion then operates on these latents, dramatically reducing computation. At inference, the denoised latent is decoded back to pixel space. Stable Diffusion uses this approach. The U-Net denoiser is conditioned on text via CLIP embeddings + cross-attention. Training cost is 150–200× lower than pixel-space diffusion. Stable Diffusion is the open-source implementation and has spawned thousands of applications.

Classifier guidance (Dhariwal & Nichol 2021) steers diffusion sampling toward a target class using a classifier p_φ(y|x_t) trained on noisy images. Modified score: ∇_x log p(x_t|y) = ∇_x log p(x_t) + s·∇_x log p_φ(y|x_t), where s is the guidance scale. Higher s → more class-specific but less diverse samples. ADM (Ablated Diffusion Model) with classifier guidance matched or beat GANs on ImageNet FID for the first time, establishing diffusion as the state of the art for image generation. Requires training a separate noisy classifier, which is inconvenient.

CFG (Ho & Salimans 2022) eliminates the need for a separate classifier by training a single conditional model alongside an unconditional model (via randomly dropping conditioning 10–20% of the time). At inference, the guided score is: ε̂_θ(x_t, y) = ε_θ(x_t, ∅) + s·(ε_θ(x_t, y) − ε_θ(x_t, ∅)), where ∅ is the null condition. Higher guidance scale s → higher fidelity to condition but lower diversity. CFG is the standard technique in all modern text-to-image systems (DALL-E 2, Stable Diffusion, Imagen, Midjourney). It dramatically improves prompt adherence and visual quality.

# CFG at inference
eps_uncond = model(x_t, t, cond=null_cond)
eps_cond = model(x_t, t, cond=text_cond)
eps = eps_uncond + guidance_scale * (eps_cond - eps_uncond)
# typically guidance_scale=7.5

DALL-E 2 (OpenAI, 2022) generates images from text via: (1) CLIP text encoder maps text to image embedding space. (2) Prior network (diffusion-based) maps text CLIP embedding to image CLIP embedding. (3) Decoder (CLIP-conditioned diffusion U-Net) generates 64×64 image from image CLIP embedding. (4) Upsampling diffusion models (to 256×256 then 1024×1024). The prior enables semantic variation — same text can produce different valid images. CFG is used throughout. DALL-E 2 produces high-quality, semantically aligned images and demonstrated CLIP's image-text embedding space is rich enough to drive high-quality generation.

Imagen (Saharia et al. 2022) uses a large pre-trained T5-XXL text encoder (frozen) to encode text prompts, then conditions a pixel-space diffusion model cascade: 64×64 base model, then 256×256, then 1024×1024 super-resolution models. Key finding: T5-XXL text encoder (4.6B params) captures rich language understanding, outperforming CLIP for text-to-image quality. Uses CFG with separate guidance weights for text and noise. Achieves SOTA on DrawBench human evaluation. Shows that language model pre-training benefits image generation — text encoder quality is as important as image decoder quality.

CLIP (Contrastive Language-Image Pretraining, Radford et al. 2021) jointly trains image and text encoders on 400M (image, caption) pairs using contrastive loss: maximize cosine similarity between matched pairs, minimize for unmatched. CLIP maps images and text to a shared embedding space. In diffusion: CLIP text embeddings condition the denoiser via cross-attention, guiding generation toward the text prompt. CLIP also enables zero-shot classification and semantic search. In Stable Diffusion, the CLIP text encoder is frozen and its output injects conditioning into every U-Net cross-attention layer.

FID (Fréchet Inception Distance) measures the distance between the real and generated image distributions in Inception-v3 feature space. It models both as Gaussians and computes: FID = ||μ_r−μ_g||² + Tr(Σ_r+Σ_g−2(Σ_rΣ_g)^{1/2}). Lower FID = better. Diffusion models have dramatically improved FID: DDPM on CelebA-HQ: FID≈3; Stable Diffusion on various benchmarks: FID<10. FID requires ~50k samples and is sensitive to: number of samples, model used (Inception-v3), image resolution, and post-processing. CLIP-FID uses CLIP features instead of Inception for better alignment with human perception.

DPM-Solver (Lu et al. 2022) derives fast ODE solvers for diffusion sampling by treating the reverse process as an ODE: dx/dt = −t s_θ(x,t). DPM-Solver uses analytical exponential integrators tailored to the diffusion ODE structure. Result: high-quality samples in just 10–20 steps (vs. DDIM's 50+ steps), with minimal quality loss. DPM-Solver++ further improves guided sampling with guided ODE formulation. These solvers are now the default in Stable Diffusion (replacing DDIM). Fast solvers made real-time generation feasible on consumer hardware, enabling applications in art, video, and real-time editing.

Diffusion models are inspired by non-equilibrium thermodynamics. The forward process (adding noise) mimics thermal diffusion — random particle motion gradually destroying structure, reaching equilibrium (maximum entropy = Gaussian noise). The reverse process (denoising) is analogous to reversing this entropy increase. Sohl-Dickstein et al. (2015) first proposed this connection, using non-equilibrium statistical physics to derive the training objective. The Fokker-Planck equation governs the evolution of probability densities under diffusion — the reverse-time SDE formulation (Song et al. 2021) makes this connection explicit.

Song et al. (2021) unified diffusion models as a continuous-time stochastic differential equation (SDE): dx = f(x,t)dt + g(t)dw (forward SDE adds noise). The reverse-time SDE: dx = [f(x,t) − g(t)² ∇_x log p_t(x)] dt + g(t)dw̄ (denoises). The score function ∇_x log p_t(x) is approximated by the trained denoising network. Different choices of f,g recover DDPM (VP-SDE), NCSN (VE-SDE), or sub-VP. This framework enables using any numerical SDE/ODE solver for sampling and connects diffusion models to score matching and Langevin dynamics in a unified theory.

Conditional generation produces samples conditioned on an input y (text, class label, image, segmentation map). Conditioning methods: (1) Cross-attention: embed y, inject at every U-Net block via cross-attention (Q from image features, K,V from text — used in Stable Diffusion). (2) AdaGN: modulate group normalization with y-dependent scale/shift. (3) Label embedding added to time embedding. (4) CFG: dropout conditioning during training, extrapolate at inference. Good conditioning requires the conditioning signal to influence generation at every spatial scale. Multi-modal conditioning (text + depth + pose) is used in ControlNet.

ControlNet (Zhang & Agrawala 2023) adds spatial control to pre-trained diffusion models (e.g., Stable Diffusion) by attaching a trainable copy of the U-Net encoder to the frozen backbone. The trainable copy processes conditioning inputs (edges, depth maps, pose keypoints, segmentation masks), and its outputs are added to the frozen backbone's features. Only the ~100M parameters of the attached copy are trained; the pre-trained backbone is frozen. This enables precise spatial control (generate an image of a person with this exact pose/edges) while preserving the pre-trained model's quality and style diversity.

Inpainting fills in masked regions of an image while being consistent with the unmasked context. Diffusion approaches: (1) Repaint (Lugmayr et al. 2022) — at each reverse step, replace unmasked regions with noisy versions of the original, sample masked regions using the model; iterate this harmony step multiple times per timestep. (2) Fine-tune with inpainting objective (Stable Diffusion inpainting) — concatenate mask and masked image as additional input channels to the U-Net. Repaint is training-free but slow; fine-tuned models are faster and more coherent. Used for photo editing, content removal, and restoration.

DDPM learns to generate images by iteratively denoising Gaussian noise — requires T=1000 sampling steps, pixel-space operation, high quality. VQ-VAE (Van den Oord 2017) encodes images to discrete codebook tokens then generates with autoregressive model (PixelCNN, Transformer) — single forward pass for encoding/decoding but autoregressive generation is slow. VQ-GAN (2021) combines VQ-VAE's discrete latents with GAN training + Transformer autoregression — used in DALL-E 1. Modern systems use VQ-VAE/KL-VAE as encoder/decoder with diffusion in latent space (Stable Diffusion) — getting the best of both: fast encoding + high-quality diffusion generation.

Text-to-video extends image diffusion to temporal sequences. Challenges: temporal consistency, computational cost (video is 3D), motion coherence. Approaches: (1) Extend U-Net with temporal 3D conv or temporal attention over frames (Imagen Video, Make-A-Video). (2) Latent video diffusion (Align-Your-Latents, VideoLDM) — compress frames temporally + spatially then diffuse. (3) DiT (Diffusion Transformer) applied spatiotemporally (Sora). Training requires large-scale video-text datasets. Key evaluation metrics: FVD (Fréchet Video Distance). Sora (OpenAI 2024) demonstrated high-quality, coherent, minute-long video generation using a DiT on video patches.

DiT (Peebles & Xie 2023) replaces the U-Net backbone in diffusion models with a Vision Transformer (ViT). Input image latent patches are tokenized; timestep and class label are embedded and injected via adaLN-Zero (zero-initialized adaptive layer normalization). Standard Transformer blocks (self-attention + FFN) process the sequence. DiT scales better than U-Net: larger DiT-XL/2 achieves better FID than U-Net ADM. DiT proved that ViT's scaling properties transfer to diffusion. Sora uses a 3D patch-based DiT for video. The DiT architecture replaced U-Net in FLUX (Stability AI's new model) and SD 3.0.

IP-Adapter (Ye et al. 2023) enables image-prompted generation by injecting CLIP image features via a decoupled cross-attention mechanism (separate K,V projection for image features added to existing text cross-attention). This allows controlling style, identity, or content from a reference image. Only the adapter's K,V projection layers (~22M params) are trained on top of frozen Stable Diffusion. At inference: text prompt controls content; reference image controls style/identity. IP-Adapter enables face-consistent generation, style transfer, and product mockups without fine-tuning the entire model.

The training objective for diffusion models is the negative ELBO: L = E[−log p_θ(x_0)] ≤ −ELBO. The ELBO decomposes as: −E[log p_θ(x_0|x_1)] + Σ_t KL(q(x_{t-1}|x_t,x_0) || p_θ(x_{t-1}|x_t)). Each KL term measures how well the learned reverse step matches the true posterior q(x_{t-1}|x_t,x_0). In DDPM, these KL terms simplify to the noise prediction MSE (up to reweighting). Ho et al. found that uniform weighting of the MSE loss (ignoring the time-dependent weight from the ELBO) gives better sample quality, trading exact likelihood bound for perceptual quality.

Prompt-to-Prompt (Hertz et al. 2022) enables text-based image editing by manipulating cross-attention maps during generation. Given a generated image and original prompt, edit by changing the prompt (e.g., add 'snowy') and inject the original image's cross-attention maps into the new generation — preserving spatial layout and structure while changing specified attributes. Three edit types: word swap, prompt refinement, attention re-weighting. No optimization or retraining required — just swap attention tensors at inference. Demonstrates that cross-attention maps encode the semantic layout of the image, providing a handle for precise semantic editing.

Diffusion models can parametrize the denoising network in three equivalent ways: (1) ε-prediction: predict added noise ε, used in DDPM — works well for low noise levels. (2) x_0-prediction: predict original denoised image — can give artifacts at high noise levels. (3) v-prediction (Salimans & Ho 2022): predict v = √ᾱ_t ε − √(1−ᾱ_t) x_0, a linear combination — balances between ε and x_0 prediction, better for high-resolution and for avoiding artifacts. v-prediction is used in Imagen, Stable Diffusion XL, and SD 3.0 for improved stability and quality, especially in high-CFG settings.

More diffusion sampling steps generally produce better samples (less discretization error in the reverse SDE/ODE) but are slower. DDPM with T=1000 steps: best quality, slowest (~30s/image). DDIM with 50 steps: good quality, faster (~3s). DPM-Solver with 20 steps: near-DDIM quality at ~1.5s. DPM-Solver++ with 10 steps: decent quality in <1s. Distillation methods (consistency models, flow matching, LCM) aim to generate in 1–4 steps. The trade-off: fewer steps = less compute but more discretization error; finding fast yet accurate solvers is an active research area.

Consistency Models (Song et al. 2023) train a model f_θ(x_t, t) that maps any noisy image x_t at any time t directly to the clean x_0 in one step — consistent along the ODE trajectory. Training: consistency distillation (uses pre-trained diffusion model as teacher) or consistency training (without teacher). Consistency models can generate images in 1–4 steps without sacrificing much quality, far faster than standard diffusion sampling. LCM (Latent Consistency Models) applies this to Stable Diffusion's latent space, enabling 4-step generation. This represents a major step toward real-time high-quality diffusion generation.

Flow Matching (Lipman et al. 2022) learns a vector field v_θ(x,t) that generates a flow transforming noise to data: dx/dt = v_θ(x,t). Training: regress v_θ on target conditional vector fields (e.g., straight paths from ε to x_0: v(x,t) = (x_0 − ε)). Compared to diffusion (curved, stochastic trajectories), flow matching uses straight-line trajectories — needs fewer steps for accurate integration (fewer NFEs). Rectified Flow is a specific instance. Stable Diffusion 3 and Flux use flow matching + MMDiT architecture. Flow matching is rapidly replacing DDPM/DDIM as the training paradigm for new image generation systems.

# Flow matching training objective
# target flow: straight path from eps to x0
t = torch.rand(batch)
x_t = t * x0 + (1-t) * eps  # interpolation
v_target = x0 - eps          # target vector field
loss = F.mse_loss(model(x_t, t), v_target)

Diffusion models have expanded to diverse modalities: (1) Audio: WaveGrad, DiffWave generate high-quality speech/music waveforms. (2) Video: Sora, Gen-2 for text-to-video. (3) 3D: DreamFusion (text-to-3D via score distillation sampling), Point-E, Shap-E for 3D point clouds/meshes. (4) Molecules: DiffSBDD for drug design (3D molecule generation). (5) Motion: MDM for human motion sequences. (6) Text: Masked diffusion for text generation. (7) Super-resolution, inpainting, restoration. The core denoising framework generalizes to any modality with appropriate data representation and network architecture.

DreamBooth (Ruiz et al. 2022) personalizes a text-to-image diffusion model to a specific subject (e.g., your dog or face) from 3–5 reference images. It fine-tunes all diffusion model weights on the reference images using a rare identifier token (e.g., 'a [V] dog') + class-prior preservation loss (generate regular dogs to prevent catastrophic forgetting). After fine-tuning (few minutes), the model generates the specific subject in novel contexts: '[V] dog on the moon'. DreamBooth enables high-quality personalized generation at low cost and spawned a cottage industry of personalization techniques (Textual Inversion, LoRA).

SDXL (2023) improves on SD 1.x in several ways: (1) Larger UNet (2.6B params vs SD1.5's 860M), with 2 CLIP text encoders (CLIP-L + OpenCLIP-G) providing richer conditioning. (2) Native 1024×1024 resolution (vs 512×512). (3) Two-stage: base model generates 1024×1024, refiner model adds detail in latent space. (4) Improved aspect ratio handling via size/crop conditioning. (5) v-prediction. SDXL is open-source and runs on consumer GPUs (8GB VRAM). SD 3.0/3.5 further improved with flow matching + MMDiT architecture, achieving near-DALL-E-3 quality in open-source form.

High-quality text-to-image diffusion models require massive training datasets. LAION-5B (5.85 billion image-text pairs scraped from the web) was used for Stable Diffusion 1.x–2.x. LAION-Aesthetics (filtered subset of highest aesthetic-scored images) is used for fine-tuning. Imagen used Google's proprietary dataset. DALL-E 3 used re-captioned data (generate high-quality captions for each image) to improve prompt alignment. Key insight: data quality matters more than quantity — filtering for aesthetic score, caption quality, and safety improves generation quality significantly. Copyright and consent issues with scraped data remain major ethical concerns.

Classifier-free guidance scale (CFG scale, s) controls how strongly the diffusion model follows the text prompt. Low CFG (s=1): ignores text, samples from unconditional distribution — diverse but random. CFG=7.5 (typical default): good balance of prompt adherence and quality. High CFG (s=15+): very prompt-adherent but oversaturated, loses fine details, artifacts. This is the main inference-time parameter users control. It works because: guided_eps = uncond_eps + s × (cond_eps − uncond_eps). Higher s amplifies the direction toward the conditional prediction, trading diversity for prompt fidelity.

Diffusion models face a fundamental quality-diversity trade-off (similar to GAN truncation). Higher CFG guidance → higher per-sample quality, lower diversity (precision vs recall in distribution-matching terms). Recall (coverage of modes) vs precision (sample quality) can be measured via recall and precision metrics on real/generated distributions. Increasing guidance scale increases precision (near-perfect images) at cost of recall (fewer modes covered). This explains why high guidance gives images that 'look like stock photos' but lack variety. Different use cases require different trade-offs: creative generation uses low CFG, product mockups use high CFG.

DDIM inversion finds the noise latent x_T corresponding to a real image x_0, enabling editing. Since DDIM is deterministic, running the forward ODE numerically gives x_T ≈ DDIM_forward(x_0). Then edit the prompt and run DDIM backward from x_T — the image retains structure from the inversion while the new prompt changes attributes. Limitation: DDIM inversion accumulates numerical error over 50 steps, leading to imperfect reconstruction. Null-text Inversion (Mokady et al. 2023) corrects this by optimizing the unconditioned embedding at each step to minimize reconstruction error, enabling high-fidelity real image editing with Prompt-to-Prompt.

The signal-to-noise ratio at time t is SNR(t) = ᾱ_t / (1−ᾱ_t), where ᾱ_t is the cumulative noise product. High SNR (small t) = mostly signal, little noise; low SNR (large t) = mostly noise. The loss weighting in DDPM assigns equal weight to all timesteps, but Min-SNR weighting (Hang et al. 2023) clips SNR at a maximum value (SNR_min=5) to balance training: at very high SNR, the loss gradient is dominated by easy high-SNR steps; clipping shifts focus to harder, noisier steps. Min-SNR weighting speeds convergence and improves sample quality, especially for high-resolution images.

The CFG scale s controls prompt adherence: ε̂ = ε_uncond + s(ε_cond − ε_uncond). Typical values: s=7.5 for Stable Diffusion (good balance), s=1 (no guidance, fully unconditional), s=15+ (hyper-detailed but can over-saturate). Higher s → image looks more like the text description but less natural, with possible artifacts and loss of fine detail. For photorealistic prompts: s=7–10. For artistic: s=5–8. For precise attribute control: s=10–15. The optimal value is task-dependent and usually found by trial. CFG comes at 2× inference cost (two forward passes: conditional + unconditional).

# Two forward passes for CFG
eps_uncond = model(x_t, t, cond=null_token)
eps_cond = model(x_t, t, cond=text_embed)
eps_guided = eps_uncond + scale * (eps_cond - eps_uncond)

In the denoising U-Net, skip connections bridge encoder (downsampling) blocks to decoder (upsampling) blocks at the same spatial resolution. They preserve fine-grained spatial details that are lost during downsampling — encoder captures local texture; decoder uses these via concatenation or addition. Without skip connections, the decoder must reconstruct fine details from only the bottleneck — blurring high-frequency content. In diffusion models at large t (heavily noisy), the U-Net mainly does global denoising; at small t, it refines fine details — skip connections are critical for this fine-detail pass.

Text-to-image alignment measures how well generated images match their text prompts. Evaluation methods: (1) CLIP score — cosine similarity between CLIP image embedding and CLIP text embedding; higher = better alignment. (2) Human evaluation — ask raters to score alignment and realism. (3) GenEval — structured benchmark testing spatial relationships, attribute binding, and object presence. (4) T2I-CompBench — compositional alignment (color+shape+texture+spatial binding). Models often score high on simple prompts but fail complex compositional ones ('a red cube to the left of a blue sphere'). This compositional binding is an active research challenge for diffusion models.

import clip, torch
image_feat = clip.encode_image(image)
text_feat = clip.encode_text(text)
score = (image_feat @ text_feat.T).item()

Textual Inversion (Gal et al. 2022) learns a new text embedding token S* for the concept by optimizing only that embedding while freezing the entire diffusion model. Only ~5 images needed; few parameters changed (~768 floats). Simple but limited expressiveness — a single token can't capture complex identities. Dreambooth fine-tunes all model weights on 3–5 images using a rare token + prior preservation loss. Far more expressive — captures complex appearances, textures, and styles. More compute and risk of overfitting. Trade-off: Textual Inversion = minimal, fast, safe; Dreambooth = high quality, more compute, some forgetting risk.

Step distillation trains a student model to produce in fewer steps what the teacher DDPM/DDIM produces in many. Progressive distillation (Salimans & Ho 2022) halves steps each round: 1000→500→...→4 steps. The student learns to predict the 2-step DDIM output in 1 step using the 2-step result as the target. Consistency Distillation trains on adjacent points on the ODE trajectory to produce consistent outputs. LCM (Latent Consistency Models) distills Stable Diffusion to 4 steps. Flash Diffusion trains a student to generate in 1–4 steps. Step distillation enables real-time generation at inference without sacrificing much quality.

The key insight in DDPM is that x_t given x_0 has a closed-form distribution: q(x_t | x_0) = N(x_t; √ᾱ_t x_0, (1−ᾱ_t)I), where ᾱ_t = Π_{s=1}^t (1−β_s). This means x_t = √ᾱ_t x_0 + √(1−ᾱ_t) ε, ε ~ N(0,I) — any timestep can be sampled in one shot without simulating t steps of the forward chain. This is the reparameterization trick for diffusion: enables training by sampling random t, adding appropriate noise, and training the denoiser. At t=T: ᾱ_T ≈ 0 so x_T ≈ N(0,I) — nearly pure Gaussian noise.

Diffusion models for super-resolution take a low-resolution (LR) image as conditioning and generate a high-resolution (HR) output. The LR image is concatenated with noisy HR latent as U-Net input. The model learns to hallucinate plausible fine details consistent with the LR image. SR3 (Saharia et al.) uses cascaded diffusion: 64→256→1024. StableSR applies latent diffusion for efficient SR. Advantages over GAN-based SR (SRGAN): more diverse outputs, fewer artifacts, better perceptual quality. Disadvantage: slower inference. Diffusion SR can add plausible textures not present in LR — important for medical imaging and satellite imagery super-resolution.

Adapting diffusion to discrete text (where standard Gaussian noise doesn't apply) requires discrete diffusion or masked diffusion. MDLM (Masked Diffusion Language Model) corrupts text by progressively masking tokens (absorbing state Markov chain) and trains to predict original tokens. D3PM uses categorical diffusion with different corruption kernels. MDLM achieves perplexity competitive with autoregressive GPT-2 on certain benchmarks, enabling parallel generation (all positions simultaneously) vs autoregressive's sequential token generation. Masked diffusion is slower to train than standard AR LMs but enables controllable, parallel generation and better editing capabilities.

The diffusion U-Net uses residual blocks (ResBlocks) as building blocks: ResBlock(x, t_emb) = x + Conv(GroupNorm(x + t_proj(t_emb))). The time embedding t_emb (from sinusoidal timestep + MLP) is added to features via a linear projection, modulating activations so the network knows which denoising step it's performing. GroupNorm normalizes over feature groups (more stable than BN for generation). Two conv layers with residual skip. Attention blocks (self-attention) are added at 8×8 and 16×16 resolution levels for global consistency. This design from OpenAI's ADM paper became the standard for pixel-space diffusion models.

Diffusion models surpassed GANs in image quality by 2022 (ADM beat GANs on ImageNet FID). Key advantages: (1) Training stability — simple MSE loss on noise prediction, no adversarial game. (2) Mode coverage — no mode collapse; diffusion covers the full data distribution. (3) Diversity — stochastic sampling produces varied outputs from same prompt. (4) Scalability — straightforward to scale with larger U-Nets/DiTs. (5) Conditioning flexibility — easy to inject conditioning (text, class, image) via cross-attention. Disadvantages: slower inference (T reverse steps vs GAN's one pass), higher compute per sample. Consistency models and flow matching reduce this gap.

The diffusion timestep t is embedded via sinusoidal encoding (analogous to positional encoding in Transformers): emb(t) = [sin(t/10000^{2i/d}), cos(t/10000^{2i/d})] for i=0,...,d/2. This sinusoidal t-embedding is projected through two linear layers with SiLU activation to produce a dense embedding vector. At each ResBlock, this embedding is added to feature maps (after linear projection) via: h = h + Linear(SiLU(Linear(t_emb))). AdaGroupNorm uses it as scale/shift parameters for normalization. This conditioning tells the network 'at this noise level, how much denoising to do and what kind of structure to produce'.

import torch, math
def sinusoidal_embedding(t, d):
    half = d//2
    freqs = torch.exp(-math.log(10000)*torch.arange(half)/half)
    args = t[:,None]*freqs[None]
    return torch.cat([torch.cos(args), torch.sin(args)], dim=-1)