Advanced Diffusion Models

DDPM defines a forward process that gradually adds Gaussian noise over T timesteps until the image becomes pure noise, then trains a neural network to reverse this process. The training objective is a variational lower bound (ELBO) on log-likelihood, which simplifies to a mean squared error loss between the true noise added at each timestep and the noise predicted by the network given the noisy image and timestep. The simplified objective—predicting epsilon (noise) rather than the data distribution directly—proved more stable and effective in practice.

DDIM reinterprets the diffusion process as a non-Markovian process with the same marginal distributions as DDPM. This allows deterministic sampling—or semi-deterministic with an eta parameter—by taking larger, consistent steps through latent space without requiring every intermediate timestep. DDPM requires T=1000 steps for high quality; DDIM achieves comparable quality in 20–50 steps. The deterministic nature of DDIM also enables latent space interpolation and image editing by inverting real images to a latent code and re-sampling with a modified prompt.

CFG trains a single conditional diffusion model that also learns the unconditional distribution by randomly dropping the condition (text prompt) during training. At inference, the effective score is: score_conditional + guidance_scale × (score_conditional − score_unconditional). Higher guidance_scale values push samples closer to the conditioning signal, increasing prompt adherence and perceived quality but reducing diversity and increasing saturation artifacts. Typical guidance scales for text-to-image are 7–9. CFG replaces the need for a separate classifier while achieving stronger conditioning than classifier guidance.

LDMs (Rombach et al., 2022—the foundation of Stable Diffusion) first train a VQ-regularized or KL-regularized autoencoder to compress images into a low-dimensional latent space (typically 64×64×4 for a 512×512 image). The diffusion U-Net then operates entirely in this compressed latent space rather than pixel space. This reduces computational cost by 64× in space and allows the denoising network to focus on semantic content, leaving pixel-level reconstruction to the decoder. Cross-attention layers in the U-Net allow text conditioning via CLIP or T5 embeddings.

DreamBooth fine-tunes a pretrained text-to-image diffusion model on 3–5 images of a specific subject, binding the subject to a rare token identifier (e.g., 'sks dog'). To prevent catastrophic forgetting and language drift, it uses a class-specific prior preservation loss: generating class images (e.g., random dog images) with the frozen model and mixing them into training to preserve the model's prior over the class. The result is a model that generates the specific subject in novel contexts, poses, and styles when prompted with the rare token.

SDXL increases the U-Net parameter count from ~860M to ~2.6B and uses a two-stage pipeline: a base model that generates a 1024×1024 latent and a smaller refiner model that adds fine detail via SDEdit-style img2img denoising in latent space. SDXL uses two text encoders (CLIP ViT-L and OpenCLIP ViT-bigG) whose embeddings are concatenated, providing richer text conditioning. Additional conditioning on original/crop/aesthetic scores during training improves output quality. The native 1024×1024 resolution dramatically improves fine detail and text rendering.

Flow matching (Lipman et al., 2022) trains a vector field that transports samples from a simple distribution (Gaussian) to the data distribution along straight-line paths (conditional flow). Unlike diffusion which uses a fixed noising process, flow matching constructs optimal transport paths, typically yielding straighter trajectories that require fewer integration steps. Rectified Flow and SD3's flow matching formulation unify with diffusion: both train a neural network on a regression objective, but flow matching uses a cleaner linear interpolation between data and noise, enabling 1–4 step sampling with excellent quality.

ControlNet duplicates the encoder layers of the frozen pretrained U-Net into a trainable copy. The trainable copy receives the spatial control signal (Canny edges, depth maps, human pose, segmentation masks) through zero-initialized convolution layers that start at zero to prevent disrupting the pretrained model. The trainable encoder outputs are added back to the locked decoder via zero-convolution connections. Only the ControlNet parameters are trained, preserving the original model's generation quality. This modular design enables precise spatial control without full fine-tuning.

SDS (Poole et al., DreamFusion 2022) enables optimization of a 3D representation (NeRF) using a pretrained 2D diffusion model as a 'prior.' At each step, the current 3D model is rendered from a random camera, the rendered image is noised, and the diffusion model predicts the gradient direction that would make this image more consistent with the text prompt. This gradient is backpropagated through the differentiable renderer to update the 3D representation. SDS avoids the need for 3D training data but suffers from over-saturation and the 'Janus problem' (multi-face artifacts), prompting improved variants like VSD and NFSD.

Inpainting fills masked regions of an image to produce a coherent result consistent with unmasked context. Diffusion inpainting models (like Stable Diffusion Inpainting) are fine-tuned with the masked image and mask concatenated as extra input channels, teaching the model to fill only the masked region while respecting unmasked pixels. Outpainting extends an image beyond its original borders by treating pixels outside the original frame as the mask. Both use the same technical approach but differ in mask geometry—inpainting uses interior masks while outpainting uses border masks requiring extrapolation.

DiT (Peebles & Xie, 2023) replaces the convolutional U-Net with a Vision Transformer (ViT) operating on latent patches. Images are tokenized into patches via a patchify layer, processed through a sequence of transformer blocks with adaptive layer norm (adaLN) conditioning on timestep and class label embeddings, then unpatchified via a linear layer. DiT achieves better FID than U-Net baselines at equivalent compute and scales predictably with model size. Stable Diffusion 3 and FLUX use DiT-based architectures (MM-DiT with separate text and image streams).

FLUX.1 (Black Forest Labs, 2024) uses a Multimodal Diffusion Transformer (MM-DiT) variant with a hybrid architecture combining bidirectional text-image attention blocks (where text and image tokens attend to each other) and single-stream blocks where they are fully mixed. FLUX uses 12B parameters in the largest variant. It applies flow matching training (same as SD3) with rotary positional embeddings (RoPE) for both image patches and text tokens. FLUX.1-dev achieves state-of-the-art image quality and prompt adherence with notable improvements in text rendering compared to earlier models.

Textual Inversion (Gal et al., 2022) keeps the entire diffusion model frozen and only optimizes a new text embedding vector (a pseudo-word) to represent the concept from 3–5 reference images. Training updates only a single embedding vector (~768 parameters) rather than millions of model weights. This makes it extremely lightweight and avoids any model alteration. DreamBooth fine-tunes all or many model parameters, capturing more nuanced subject details at the cost of storage (one full model per concept). Textual Inversion embeddings are fully composable with other learned concepts in the same prompt.

Denoising Score Matching trains a model to estimate the score function (gradient of log data density) from noisy observations. By adding noise of known variance to data samples and training the model to predict the score of the noisy distribution, DSM provides a scalable approximation to score matching. DDPM's epsilon-prediction objective is equivalent to scaled DSM across noise levels. Song et al.'s score-based SDE framework unified DSM and diffusion: the noising process is an SDE and the denoising is its reverse, with the score network learned via DSM enabling flexible diffusion schedule design.

Video diffusion models extend image diffusion to generate temporally coherent frame sequences. Architectures add temporal attention or 3D convolutions to the U-Net or DiT, allowing frames to attend to adjacent frames. The main challenges are: (1) temporal consistency—objects must maintain identity across frames; (2) motion smoothness—avoiding flickering; (3) computational cost—O(T×H×W) for T frames; (4) long-range consistency in longer videos. Models like ModelScope, Sora, and CogVideoX use temporal attention + spatial attention or full 3D attention with aggressive compression to manage compute.

Consistency Models (Song et al., 2023) train a function that maps any point on a diffusion trajectory directly to the trajectory's origin (clean data). The consistency condition enforces that all points on the same trajectory map to the same clean output. Training uses either distillation from a pretrained diffusion model (Consistency Distillation) or direct training from scratch (Consistency Training). After training, any amount of computation can be traded off: single-step for speed or multi-step for quality by renoising partially and applying the model again. LCM (Latent Consistency Models) applies this to the latent space of Stable Diffusion.

LoRA adapts pretrained diffusion model weights by decomposing weight updates into low-rank matrices: ΔW = A×B where A ∈ R^(d×r) and B ∈ R^(r×k), with rank r << min(d,k). Only A and B are trained during fine-tuning; the original weights remain frozen. For diffusion models, LoRA is applied to the U-Net attention layers (Q, K, V, Out projections). A typical LoRA rank of 4–64 reduces trainable parameters by 100–1000×. Multiple LoRAs can be merged at inference with adjustable weights, enabling style mixing without retraining. LyCORIS extends LoRA with additional matrix decompositions (LoCon, LoHa).

# LoRA weight decomposition example
rank = 4
import torch.nn as nn
# Replace Linear(d, k) with:
lora_A = nn.Linear(d, rank, bias=False)
lora_B = nn.Linear(rank, k, bias=False)
# delta_W = lora_B(lora_A(x)) * scale

Classifier guidance (Dhariwal & Nichol, 2021) requires training a separate noise-conditioned classifier p(y|x_t) on noisy images, then using its gradient ∇_{x_t} log p(y|x_t) to steer the diffusion process toward high-class-probability regions. It improves FID but requires a separate classifier trained specifically on noisy data. Classifier-free guidance eliminates this requirement by training a single model to handle both conditional and unconditional generation. CFG is simpler, avoids the noisy classifier training challenge, and has become the universal standard, while classifier guidance is now mainly of historical interest.

High-resolution generation (e.g., 2048×2048) is expensive because self-attention in DiT scales quadratically with the number of patches. Perceiver AR introduces a set of latent tokens that cross-attend to image patches, reducing the sequence length on which self-attention operates from H×W patches to a fixed number of latents. This enables O(L×N + L²) complexity instead of O(N²) where L (latents) << N (patches). Combined with diffusion, this allows generating very high-resolution images without the quadratic blowup of standard transformer architectures.

Negative prompting provides an alternative text condition that the model should avoid. In CFG, the unconditional score is replaced by the negative-prompt-conditioned score: final_score = score(positive_prompt) + guidance_scale × (score(positive_prompt) − score(negative_prompt)). This pushes generation toward positive features while pushing away negative features. Common negative prompts include 'blurry, deformed, low quality, watermark.' Negative prompting is computationally free as it replaces the unconditional forward pass with a negative prompt forward pass, using the same two inference passes per step.

IP-Adapter (Ye et al., 2023) adds image conditioning to a frozen diffusion U-Net by inserting a lightweight decoupled cross-attention layer alongside each existing text cross-attention layer. The reference image is encoded by CLIP image encoder; the features are projected and fed into the new cross-attention modules. The original text cross-attention layers remain unchanged and fully functional. Only the new adapter's parameters (~22M) are trained. At inference, image prompt features and text prompt features are added via separate attention layers, allowing simultaneous text and image conditioning.

SDXL-Turbo (Sauer et al., 2023) applies Adversarial Diffusion Distillation (ADD): it distills a 40-step SDXL teacher into a student that generates high-quality images in 1–4 steps. ADD combines score distillation (using the teacher as a 'critic') with a discriminator adversarial loss to prevent mode collapse and improve perceptual quality at low step counts. The discriminator is initialized from a pretrained DINO encoder for strong feature-level discrimination. SDXL-Turbo achieves real-time 512×512 generation (~100ms on a single A100) with competitive image quality.

FID (Fréchet Inception Distance) measures distributional similarity between generated and real image feature statistics using Inception-v3 activations—lower is better. CLIP Score measures alignment between generated image and text prompt via CLIP cosine similarity—higher is better. Aesthetic score uses a LAION-trained aesthetic quality classifier. Human evaluation via Elo ratings or preference studies remains the gold standard. IS (Inception Score) is now less used. For prompt adherence specifically, DrawBench and T2I-CompBench test compositional prompts requiring attribute binding and spatial reasoning.

The noise scheduler defines the variance schedule—how much noise is added at each timestep. DDPM uses a linear schedule from β₁=0.0001 to βT=0.02. The improved DDPM uses a cosine schedule that reduces noise more gradually at the ends, preventing near-zero signal at very early steps. Zero-terminal SNR schedules (Lin et al., 2023) force the final timestep to pure noise, resolving a mismatch that caused color shifts. Flow matching uses a linear interpolation schedule. Schedule choice affects training stability, sample diversity, and the number of steps needed for good quality.

The Janus problem occurs in score distillation sampling for 3D generation: the diffusion prior trained on 2D images often generates a face-like structure on multiple sides of a 3D object (e.g., a 3D character with a face visible from all angles). This happens because the 2D prior has no global 3D consistency—it simply generates what looks like a plausible 2D view from each camera angle independently. Solutions include: Perp-Neg that negates the gradient component perpendicular to the front view, viewpoint-conditioned text prompts ('front view of...'), and using 3D-aware generative models as the prior instead.

Prompt-to-Prompt (Hertz et al., 2022) performs text-guided image editing by injecting attention maps from the source generation into the editing process. First, the source image is generated and all spatial attention maps are saved. During re-generation with the edited prompt, the spatial layouts from source attention maps are injected, preserving structure while the text change alters appearance. This enables localized edits (changing 'cat' to 'dog' while preserving pose), global style edits, and attention-based amplification of specific words—all without mask creation or additional training.

Diffusion-based super-resolution conditions the denoising U-Net on the low-resolution input image, concatenated with the noisy high-resolution estimate as additional channels. The model learns to hallucinate realistic high-frequency detail consistent with the low-resolution input. SR3 (Google) trained a U-Net conditioned on bicubic-upsampled low-res images and achieved state-of-the-art perceptual quality. Stable Diffusion's img2img with high-res fix also functions as super-resolution. These models may hallucinate details not in the original, creating a fidelity vs. perception tradeoff tracked by LPIPS and PSNR metrics.

The VAE encoder compresses 512×512 RGB images (3×512×512=786K values) to a 4×64×64=16K latent, achieving ~49× compression. The diffusion process operates on these latents. The VAE decoder reconstructs pixel images from diffusion outputs. If the VAE decoder has low capacity or poor training, artifacts manifest as: blurry fine details, color fringing, tile-like patterns, or degraded text. SDXL's improved VAE (trained with more data and EMA weights) significantly reduced blue color casts and sharpened fine details compared to SD 1.x. Using FP32 or BF16 for VAE decoding avoids numerical precision artifacts.

AnimateDiff inserts a pretrained temporal attention module (a motion module) into each layer of a frozen image diffusion U-Net. The motion module learns general motion priors from video datasets by attending across the frame dimension. During inference, the frozen spatial layers generate per-frame coherent content while the temporal attention ensures frame-to-frame consistency. Because the spatial layers remain frozen, any LoRA or fine-tuned model checkpoint can be animated by combining it with the motion module. This enables personalized video generation without video-specific fine-tuning of each model.

RLHF for image generation (e.g., ImageReward, HPSV2, PICKSCORE) trains a reward model on human preference data comparing pairs of generated images for the same prompt. The reward model scores image quality and prompt adherence. This reward is then used to fine-tune the diffusion model via RL: generating images, scoring them with the reward model, and using the score to update denoising network parameters via policy gradient or reward-weighted regression. DDPO (Denoising Diffusion Policy Optimization) frames the diffusion process as a multi-step MDP amenable to PPO.

Unconditional diffusion models generate images purely from random noise without any input conditioning—they model the full data distribution and produce random samples from learned classes or domains (e.g., all faces, all landscapes). Conditional models generate images guided by a conditioning signal: class label, text prompt, image, mask, or sketch. Conditioning is implemented via cross-attention with text embeddings, class embeddings added to timestep embeddings (DDPM with class conditioning), or input concatenation. Most production models are conditional; unconditional models are mainly used for studying distribution properties.

MultiDiffusion tiles the denoising process across overlapping image windows. At each timestep, the model denoises each tile independently, then merges the predictions in overlapping regions by averaging. This allows generation at arbitrary resolutions and aspect ratios using a model trained at fixed resolution (e.g., 512×512). For panoramas, additional region-specific prompts can guide each tile, producing coherent wide-format images. The approach works because diffusion denoising is locally consistent—patches can be independently denoised and globally consistent when properly blended.

Instruct-Pix2Pix (Brooks et al., 2023) fine-tunes Stable Diffusion on a dataset of (original image, editing instruction, edited image) triplets generated by combining GPT-4 (for instructions) and Prompt-to-Prompt (for image pairs). At inference, the original image is encoded to latent space, mixed with noise, and denoised conditioned on both the original image (via concatenation) and the editing instruction (via cross-attention with CFG). The model learns to apply semantic edits described in natural language globally, without requiring the user to draw masks or specify regions.

SDXL is trained natively at 1024×1024. If images larger than this are generated directly, tiling artifacts and content repetition emerge because the model has never seen such resolutions during training. The high-res fix generates a low-resolution latent first (e.g., 512×512), then upscales it with bilinear interpolation, and applies a small number of diffusion steps (img2img with low denoising strength ~0.4–0.7) to add high-frequency detail. This leverages the fact that diffusion models at low noise levels add fine texture, effectively super-resolving the initial coarse composition.

V-prediction (Salimans & Ho, 2022) trains the diffusion model to predict v = α_t × epsilon − σ_t × x_0 (a weighted combination of noise and clean data), derived from the signal-to-noise velocity of the forward process. Compared to epsilon-prediction, v-prediction is more numerically stable at low noise levels (high signal-to-noise ratio), where epsilon-prediction gradients vanish. This makes v-prediction beneficial for: training at full resolution (where later timesteps have very low noise); progressive distillation of diffusion models; and consistency model training. SDXL and Stable Diffusion 2.x use v-prediction.

Sora (2024) uses a visual encoder to compress videos into lower-dimensional spacetime patches—treating video as a sequence of 3D patches in a spatial-temporal grid rather than separate frames. These patches are processed by a DiT-based transformer that performs full spacetime attention across all patches simultaneously. Variable-length, variable-resolution, variable-duration videos are handled natively by adjusting patch counts. Text conditioning is provided via T5 embeddings with CFG. Training on internet-scale video data with detailed recaptioning (using GPT-4V to generate dense captions) is central to Sora's emergent physical simulation capability.

Cross-attention maps in diffusion U-Nets show which spatial regions correspond to each text token at each denoising step. Visualizing these maps reveals prompt-image alignment issues: if 'blue car' produces attention maps where 'blue' activates over the sky region, the model is incorrectly binding attributes. Attention map analysis helped motivate Attend-and-Excite and Divide-and-Conquer approaches. Self-attention maps reveal object segmentation structure. At early timesteps, cross-attention maps establish coarse composition; at later steps, self-attention refines local texture, making the maps useful for understanding generation dynamics and diagnosing compositional failures.

LCM-LoRA (Luo et al., 2023) distills a Latent Consistency Model into a small LoRA adapter (~64MB) that can be applied to any Stable Diffusion 1.5 or SDXL checkpoint. The LoRA adapter is trained on the consistency model objective against the original SD model as teacher. At inference, merging this LoRA transforms any base model (including custom fine-tunes and style models) into a 2–8 step generator without retraining the entire model. The adapter is model-specific (SD1.5 vs SDXL) but checkpoint-agnostic within each model family.

CLIP text encoders (up to 77 tokens) are trained with contrastive image-text objectives, giving them strong semantic-visual alignment but limited capacity for complex compositional language. T5 (up to thousands of tokens) is trained on pure text with seq2seq language modeling, giving it better understanding of complex prompts, rare words, negations, and long descriptions. SDXL uses dual CLIP encoders; SD3 and FLUX use a combination of CLIP and T5-XXL (4.7B params). T5 substantially improves following of complex compositional prompts and long descriptions at the cost of T5's large memory footprint.

DDIM inversion reverses the deterministic DDIM sampling process to find the noise latent z_T from which a real image would be generated. Starting from a real image encoded to z_0, the forward DDIM process is run in reverse (using the denoising network) to estimate z_T. This inverted latent can then be re-denoised with a modified prompt to edit the image while preserving structure. Limitations include: (1) inversion error accumulation over T steps; (2) the approximation works only for small CFG guidance during inversion; (3) editing changes often leak into unintended regions; (4) complex backgrounds are not perfectly preserved.

Commercial T2I models use a mix of: (1) high-quality curated image-text pairs (stock photos with professional captions); (2) web-scraped pairs filtered by aesthetic score, CLIP similarity, and safety classifiers; (3) synthetic data with model-generated captions recaptioning existing datasets (improving caption accuracy vs. noisy alt-text); (4) domain-specific data for rare concepts; (5) preference data for RLHF fine-tuning. Recaptioning with a VLM (e.g., LLaVA or Qwen-VL) to generate dense descriptions from images significantly improves prompt following by providing richer training signal than original alt-text.

Standard diffusion models struggle with compositional prompts like 'a red cube and a blue sphere' because cross-attention allows attribute leakage across objects. Techniques include: (1) Attend-and-Excite—maximizing attention for all subject tokens during inference; (2) structured diffusion—using linguistic structure (dependency parsing) to separate attribute-object pairs in attention; (3) regional prompting—assigning different prompts to spatial regions; (4) layout-conditioned generation (GLIGEN) where bounding box annotations guide object placement. These approaches improve attribute binding from roughly 60% to 75–85% on compositional benchmarks.

Progressive Distillation (Salimans & Ho, 2022) trains a student model to predict in one step what the teacher model predicts in two steps, then uses the student as the new teacher and repeats. After K rounds of halving, the original T-step model is distilled to a T/2^K-step model. Applied to Stable Diffusion, 1000 steps can be distilled to 4–8 steps with acceptable quality loss. Unlike consistency distillation which targets clean data, progressive distillation matches intermediate denoising targets, making it more stable and less prone to mode collapse. V-prediction parameterization is used for stability.

Safety layers in commercial T2I deployments include: (1) training data filtering—removing CSAM and extreme NSFW using classifiers (e.g., CLIP-based NSFW detectors) before training; (2) concept erasure (FMN, ESD)—fine-tuning the model to unlearn specific concepts like celebrity faces or explicit content; (3) inference-time filtering—running safety classifiers (stable-diffusion-safety-checker) on generated images before delivery; (4) prompt filtering—blocking or rewriting prompts containing prohibited terms; (5) negative prompt injection—automatically adding quality/safety negative prompts. Multiple defense layers are used since no single layer is fully robust.

Stochastic DDIM (eta > 0) adds Gaussian noise at each step proportional to eta, making the trajectory non-deterministic—different random seeds produce different images even from the same z_T. At eta=1, DDIM recovers DDPM's stochastic sampling. Deterministic DDIM (eta=0) produces exactly the same image for the same z_T, prompt, and seed, enabling: reproducible generation, interpolation between images (by interpolating latents z_T), and DDIM inversion. The stochastic variant can escape local minima for higher diversity; the deterministic variant is essential for editing applications.

U-Net skip connections concatenate encoder feature maps directly to decoder feature maps at matching resolutions. These connections enable the decoder to access fine spatial details from the encoder without bottleneck compression, crucial for preserving high-frequency textures, edges, and patterns. Without skip connections, the bottleneck would lose spatial information and outputs would be blurry. In diffusion U-Nets, skip connections at the highest resolution layers are most critical for fine detail; at lower resolutions, they pass structural information. Self-attention at skip connection levels allows global context to modulate local features before passing to the decoder.

SDXL (2023): dual CLIP text encoders, U-Net backbone, 2.6B params, strong for style fine-tuning with LoRA, large ecosystem of checkpoints. SD3 (2024): MM-DiT with combined text-image streams, T5+CLIP dual encoder, flow matching, 2B–8B variants, improved text rendering and prompt following, fewer LoRA ecosystems. FLUX.1 (2024): 12B MM-DiT hybrid with bidirectional and single-stream blocks, T5+CLIP, flow matching, state-of-the-art quality/text rendering, higher VRAM requirement. SDXL best for ecosystem/fine-tuning; SD3 for balance; FLUX.1 for maximum quality when VRAM is not a constraint.