Cheat sheet · LLMs & Agents

Generative AI cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

A dense, interview-grade reference on modern generative modeling centered on diffusion (DDPM/DDIM, score-based SDEs, latent diffusion) and flow matching, plus sampling, guidance, conditioning, evaluation, and the 2024-2026 state of the art for image, video, and audio.

Generative Landscape (where diffusion fits)

Likelihood-based: autoregressive (exact NLL, slow sequential sampling), VAEs (latent + ELBO, tend to blur), normalizing flows (exact density via invertible maps, architecturally restrictive).
Implicit/adversarial: GANs are fast (1-step) and sharp but unstable to train and prone to mode collapse; no tractable likelihood.
Diffusion / score-based: train a denoiser to model the data score. Trade slow iterative sampling for stable training, strong mode coverage, and SOTA fidelity.
Flow matching / rectified flow: learn a velocity field along (often straight) probability paths. Same backbone, simpler regression objective, fewer steps — dominant in 2024+ (SD3, Flux, Movie Gen).
Mental model: diffusion = gradually destroy data with noise (forward), then learn to reverse it (generation). It is a hierarchical latent-variable model with a fixed forward process and a learned reverse process.

DDPM: forward & reverse process

Forward is a fixed Markov chain adding Gaussian noise over T steps with variance schedule beta_t; it has no learnable parameters.
Closed-form jump to any step with alpha_bar_t = prod_{s<=t}(1-beta_s): x_t = sqrt(alpha_bar_t)*x0 + sqrt(1-alpha_bar_t)*eps, eps~N(0,I).
Reverse p_theta(x_{t-1}|x_t) is a learned Gaussian; the network typically predicts the noise eps_theta(x_t, t), and the posterior mean is derived from it.
Simplified loss (Ho 2020): L_simple = E_{t,x0,eps} ||eps - eps_theta(x_t, t)||^2 — a denoising objective that is a reweighted variational bound (drops the ELBO weighting term).
Parameterizations: predict eps (noise), x0 (clean sample), or v = sqrt(alpha_bar_t)*eps - sqrt(1-alpha_bar_t)*x0 (v-prediction — well-behaved across SNR, used for high-res and distillation).
Schedules: linear (original) vs cosine (Nichol & Dhariwal) which avoids over-noising small images; for high resolution shift the schedule / SNR and apply zero-terminal-SNR to fix the washed-out-brightness bug.

Score-based view & SDE/ODE unification

Score = gradient of log-density: s_theta(x,t) ~= grad_x log p_t(x). Denoising score matching shows eps-prediction equals learning the score up to scale: score = -eps_theta / sigma_t (sigma_t = sqrt(1-alpha_bar_t) in VP).
Song et al. unify diffusion as an SDE: forward dx = f(x,t)dt + g(t)dw; VP-SDE is continuous DDPM, VE-SDE is continuous SMLD/NCSN.
The reverse-time SDE uses the score; there is an equivalent deterministic probability-flow ODE with the same marginals — enabling fast, near-deterministic sampling and exact likelihood via the instantaneous change-of-variables.
SDE samplers inject fresh noise each step (diverse, error-correcting); ODE samplers (DDIM, DPM-Solver, Euler) are deterministic and reusable for inversion and editing.
Flow matching trains a velocity v_theta by regressing on a (data,noise) interpolant; rectified flow straightens trajectories so few-step / 1-step sampling works well.

Sampling, guidance & acceleration

DDIM: non-Markovian, deterministic at eta=0; good quality at 20-50 steps vs ~1000 for ancestral DDPM. DPM-Solver / DPM-Solver++ (high-order ODE solvers) give strong samples in ~10-20 steps.
Classifier guidance: add grad of a noise-aware classifier (needs a separate model). Classifier-Free Guidance (CFG) is the standard: jointly train conditional + unconditional by dropping the condition ~10% of the time, then extrapolate at inference.
CFG: eps_hat = eps_uncond + w*(eps_cond - eps_uncond). Higher w improves prompt adherence but increases saturation/artifacts and cuts diversity; CFG runs two forward passes per step (~2x inference cost).
Few/1-step distillation: progressive distillation, consistency models / LCM, and adversarial distillation (SDXL-Turbo, SD3-Turbo). Consistency models map any x_t directly to x0 along an ODE trajectory.
Practical knobs: scheduler, step count, CFG scale, negative prompts, seed; CFG-rescale or dynamic thresholding tame over-saturation at high guidance.

Latent diffusion, conditioning & architecture

Latent Diffusion (Stable Diffusion): run diffusion in a VAE latent (e.g. 8x spatial downsample) — ~64x fewer spatial elements, slashing compute while keeping fidelity; the key enabler of consumer text-to-image.
Backbones: U-Net (ResNet blocks + self/cross-attention) historically; Diffusion Transformers (DiT) now scale better and underpin SD3, PixArt, and Sora-class video models.
Conditioning: text via cross-attention from CLIP/T5 encoders; timestep via sinusoidal embeddings; injection via AdaLN/AdaLN-Zero (DiT) or FiLM; class via learned embeddings.
Control & personalization: ControlNet (edges/pose/depth), T2I-Adapters, IP-Adapter (image prompts); DreamBooth, LoRA, and textual inversion for subject/style finetuning.
Modalities: video adds temporal attention + 3D/causal VAEs (Sora, Movie Gen, Veo); audio uses mel-spectrogram or waveform diffusion (Stable Audio); 3D via SDS / Gaussian-splat pipelines.

Evaluation & deployment realities

Image metrics: FID (feature-distribution distance, lower is better — sensitive to sample count/preprocessing/backbone), Inception Score, CLIPScore for text alignment, Precision/Recall for fidelity vs diversity.
FID is biased and gameable; modern work also reports human preference (e.g. ELO/PartiPrompts) and CLIP/DINO-based metrics. For video add temporal-consistency and motion scores (e.g. FVD).
Cost: each sample = (steps x CFG-passes) forward calls. Reduce via latent space, high-order solvers, distillation, fp16/bf16, flash/fused attention kernels, and batching.
Safety/IP: NSFW and concept filters, invisible watermarking (SynthID) and provenance (C2PA), dataset provenance, and copyright/likeness concerns are now first-class deployment requirements.
Trends (2026): flow-matching + DiT as default; unified multimodal models with native autoregressive image generation; real-time and 1-4 step generation; longer coherent video; on-device distilled models.

Common pitfalls

Confusing the forward process (fixed, no learning) with the reverse process (the learned generator).
Saying the model 'predicts the image' — most DDPMs predict the noise eps (or v); the clean sample x0 is reconstructed from it.
Treating score-based models and DDPMs as different — they are the same framework (score = -eps/sigma_t); the SDE view unifies them.
Thinking higher CFG is always better; large guidance over-saturates, reduces diversity, and adds artifacts — needs CFG-rescale/dynamic thresholding.
Forgetting CFG roughly doubles inference cost because it runs both conditional and unconditional passes per step.
Comparing FID across papers without matching sample count, resolution, feature backbone, and preprocessing — FID is biased and not comparable out of the box.
Assuming you need ~1000 steps; DDIM/DPM-Solver and distilled/consistency models reach good quality in 1-50 steps.
Ignoring zero-terminal-SNR / schedule-shift, which causes washed-out brightness and high-res quality loss.
Conflating latent diffusion's VAE (perceptual compression) with the diffusion model itself.

Interview tips

Write the forward closed-form (x_t from x_0) and the simplified DDPM loss from memory.
Explain why diffusion trains stably vs GANs (regression objective, no minimax, full mode coverage) and the cost (iterative sampling).
Articulate the score / SDE / probability-flow-ODE connection and how DDIM arises as the deterministic ODE solver.
Derive classifier-free guidance, the eps_uncond + w*(eps_cond - eps_uncond) formula, and the quality/diversity/cost trade-off.
Discuss how to make sampling fast: high-order solvers, distillation, consistency models, latent space, flow matching.
Know latent diffusion and DiT vs U-Net, plus conditioning mechanisms (cross-attention, AdaLN, ControlNet/LoRA/IP-Adapter).
Compare diffusion vs flow matching / rectified flow and why FM dominates recent SOTA systems (SD3, Flux).
Talk through evaluation honestly: FID limitations, CLIPScore, human preference, and video temporal metrics (FVD).
Bonus signal: mention watermarking/provenance (SynthID, C2PA), copyright, and real-time/on-device deployment constraints.

Read the full lesson Test yourself — 25 question quiz