GANs

A GAN has a Generator G and a Discriminator D. The generator takes random noise z ~ p_z(z) and produces synthetic data G(z). The discriminator takes real or fake samples and outputs a probability that the input is real: D(x) ∈ [0,1]. G tries to fool D; D tries to correctly classify real vs. fake. They are trained adversarially: G improves to produce more realistic samples, and D improves to better detect fakes. At convergence, G(z) ≈ p_data(x).

The GAN objective is a minimax game: min_G max_D V(D,G) = E[log D(x)] + E[log(1 − D(G(z)))]. D maximizes this (correctly classify real and fake); G minimizes it (fool D into outputting high values for G(z)). The optimal discriminator given G is D*(x) = p_data(x) / (p_data(x) + p_G(x)). At the global optimum, p_G = p_data and D outputs 1/2 everywhere. In practice this exact optimum is rarely reached; training is an iterative approximation.

Early in training, D easily distinguishes real from fake (D(G(z)) ≈ 0). The generator's original loss log(1−D(G(z))) saturates near 0 gradient. Fix: use the non-saturating generator loss −log D(G(z)) instead. When D is confident the sample is fake, −log D(G(z)) is large, giving strong gradients to G. This practical modification is used in almost all GAN implementations, though it changes the minimax equilibrium to a heuristic one.

# Non-saturating generator loss
g_loss = -torch.log(D(G(z))).mean()

Mode collapse occurs when the generator learns to produce only a few types of outputs — mapping many noise vectors z to the same output — rather than capturing the full diversity of p_data. For example, a GAN trained on MNIST might only generate '3's. The generator exploits a weakness in D by repeating the samples D can't distinguish as fake. Solutions include minibatch discrimination (D compares samples in a batch), unrolled GANs, Wasserstein GANs, and spectral normalization. Mode collapse is one of the main training failure modes.

WGAN replaces the Jensen-Shannon divergence implicit in the standard GAN with the Wasserstein-1 (Earth Mover's) distance: W(p_r, p_g) = sup_{||f||_L≤1} E[f(x)] − E[f(G(z))]. The discriminator becomes a 'critic' (no sigmoid) trained to be 1-Lipschitz (enforced via weight clipping or gradient penalty). WGAN provides meaningful gradients even when p_r and p_g don't overlap, avoiding vanishing gradients and mode collapse. The WGAN loss correlates with sample quality, making training more interpretable.

DCGAN (Deep Convolutional GAN) applies convolutional networks to image generation. Key design rules: use strided convolutions (no pooling) in D, transposed convolutions in G; batch normalization in both (except D's input and G's output); LeakyReLU in D, ReLU in G, Tanh at G's final output. These rules stabilize training and produce high-quality images. DCGAN was a turning point showing that carefully designed CNNs could train stably as GANs and learn interpretable latent space features.

A conditional GAN conditions both G and D on extra information y (e.g., class label). G generates G(z,y) — a sample conditioned on y; D discriminates whether (x,y) pairs are real. This allows controlled generation: ask for a specific class, style, or attribute. Examples: Pix2Pix (image-to-image translation), label-to-image synthesis. The conditioning can be via concatenation of y with noise/feature maps or via projection. cGANs are far more useful than unconditional GANs for most practical applications.

In game theory, a Nash equilibrium is where no player can improve by unilaterally changing strategy. For GANs, the equilibrium is p_G = p_data (generator matches the data distribution) and D* = 1/2 everywhere (discriminator can't tell real from fake). Training alternates gradient steps for G and D; ideally they converge to this equilibrium. In practice, this is hard to reach because the GAN objective is non-convex and training is not guaranteed to converge — oscillations and mode collapse are common.

IS measures two things: (1) Quality — generated images should be classifiable (low entropy p(y|x)); (2) Diversity — the marginal p(y) = ∫ p(y|x) p_G(x) dx should be high entropy. IS = exp(E_x[KL(p(y|x)||p(y))]). Higher IS indicates sharper, more diverse images. Limitations: IS uses an Inception classifier trained on ImageNet, so it doesn't detect memorization, measures diversity only for ImageNet classes, and can be gamed. FID is now preferred because it's more robust and correlates better with human perception.

FID measures the distance between the distribution of real images and generated images in the feature space of an Inception network. Both sets are modeled as Gaussians with mean μ and covariance Σ: FID = ||μ_r − μ_g||² + Tr(Σ_r + Σ_g − 2(Σ_r Σ_g)^{1/2}). Lower FID = better. FID captures both quality and diversity, is sensitive to mode collapse, and correlates well with human judgment. It requires ~50k samples for reliability. FID is the standard GAN evaluation metric.

The latent space z is a low-dimensional prior distribution (typically Gaussian or Uniform) from which the generator samples to create data. The generator G: z → x maps latent vectors to data space. Directions in z can encode meaningful features — e.g., in face GANs, interpolating between two z vectors smoothly blends facial attributes. Arithmetic in z space works: z_king − z_man + z_woman ≈ z_queen. The latent space compactly represents the data manifold, and its structure reveals what the generator has learned.

# Interpolate between two latent vectors
z_interp = alpha * z1 + (1 - alpha) * z2
generated = G(z_interp)

Spectral normalization constrains each weight matrix W in the discriminator so its largest singular value (spectral norm) equals 1: W_SN = W / σ(W). This enforces the 1-Lipschitz condition required by WGAN without weight clipping, which can limit model capacity. Spectral normalization is computed efficiently using power iteration. It stabilizes GAN training across many architectures (SNGAN) without hyperparameter sensitivity. Unlike batch norm, spectral normalization applies to weights not activations, and works naturally with any layer type.

Both are generative models, but differ fundamentally. VAE (Variational Autoencoder) learns a structured latent space by encoding inputs to distributions and optimizing a lower bound on log-likelihood. VAEs produce blurry images but have meaningful, continuous latent spaces. GANs use adversarial training without explicit likelihood, producing sharper images with better perceptual quality but harder training and less interpretable latent spaces. VAEs are better for reconstruction/interpolation; GANs for perceptual quality. VQVAE and diffusion models have largely superseded both for high-quality generation.

ProGAN trains GANs by starting at low resolution (4×4) and progressively adding layers to both G and D to grow resolution (8×8, 16×16, ... 1024×1024). At each stage, new layers are faded in smoothly via a weighting parameter α. This stabilizes training because low-resolution images are easy to model, and the network learns coarse structure before fine details. ProGAN produced the first high-quality 1024×1024 face images and was a major milestone in GAN development, influencing StyleGAN.

StyleGAN (Karras et al. 2019) maps the latent z through a mapping network to an intermediate latent w, then injects w into each convolutional layer via Adaptive Instance Normalization (AdaIN). 'Style' refers to the scale and shift parameters of AdaIN that modulate feature maps. Different layers control different scales: coarse layers (low resolution) control pose/shape; fine layers control color/texture. This disentangles style control at different scales and enables style mixing. StyleGAN2/3 further removed artifacts and enabled better training.

Minibatch discrimination lets the discriminator compare samples within a mini-batch, helping detect mode collapse. The discriminator computes pairwise feature similarities across the batch and includes these as extra features. If G is collapsing (many near-identical samples), D can detect the low diversity within the batch and penalize G. Without this, D evaluates each sample independently and can be fooled by a generator producing one good but repetitive sample. Minibatch discrimination was introduced in the DCGAN paper and improves diversity.

If D is too strong, G gets vanishing gradients (D(G(z)) ≈ 0, log(1−D(G(z))) ≈ 0). If G is too strong, D can't distinguish real from fake, providing no useful signal. Practitioners typically run more D updates per G update (e.g., 5 D steps per G step in WGAN) or carefully tune learning rates. If D learns too fast, use lower D learning rate or add dropout to D. Signs of imbalance: D loss → 0 (D wins), or G loss → 0 (G wins). Training monitoring is essential.

Standard GAN training alternates: (1) Sample real data x ~ p_data, fake data z ~ p_z, compute G(z). (2) Update D: maximize log D(x) + log(1−D(G(z))) — i.e., minimize cross-entropy for real/fake classification. (3) Sample new z ~ p_z. (4) Update G: minimize log(1−D(G(z))) or equivalently maximize log D(G(z)) (non-saturating). Repeat. Typically D is updated more often per G step. Gradients do not flow through D when updating G for step 4 in most implementations.

for real in dataloader:
    # Train D
    z = torch.randn(bs, latent_dim)
    d_loss = -log(D(real)).mean() - log(1-D(G(z))).mean()
    # Train G
    g_loss = -log(D(G(z))).mean()

Pix2Pix (Isola et al. 2017) uses a conditional GAN for paired image-to-image translation: given source image x, generate target y (e.g., sketch→photo, day→night, segmentation→street). The generator is a U-Net; the discriminator is a PatchGAN that classifies NxN patches as real/fake rather than the whole image, focusing on local texture realism. The loss combines adversarial loss (GAN) + L1 reconstruction loss (pixel-level fidelity). Pix2Pix requires paired training data; CycleGAN removes this requirement.

CycleGAN enables unpaired image-to-image translation (e.g., photos↔paintings) without paired training data. It trains two generators G: X→Y and F: Y→X plus two discriminators D_X and D_Y. The key is the cycle consistency loss: F(G(x)) ≈ x and G(F(y)) ≈ y. This prevents mode collapse and ensures G and F are consistent inverses. Adversarial losses make outputs look like the target domain. CycleGAN works surprisingly well for style transfer, season conversion, and domain adaptation.

PatchGAN classifies overlapping NxN patches of an image as real or fake, rather than the entire image. The output is a grid of real/fake decisions averaged for the final D loss. This focuses D on local texture and style rather than global coherence (which the G's L1/L2 loss handles). PatchGAN is fully convolutional, runs faster, and has fewer parameters than a global discriminator. Patch size N controls scale of texture modeled: small N = fine texture, large N = coarser structure.

Label smoothing replaces hard labels 1.0/0.0 with soft targets like 0.9/0.1 when training D. Real samples are labeled 0.9 instead of 1.0; fakes labeled 0.1 instead of 0.0. This prevents D from becoming overconfident, leaving larger gradients for G and improving training stability. It's a standard regularization trick from classification that transfers naturally to GANs. One-sided label smoothing (only smooth real labels, keep fake at 0) is often preferred because smoothing fake labels can reduce D's ability to reject bad samples.

real_labels = torch.ones(bs) * 0.9  # one-sided smoothing

BigGAN scales up GAN training to large batch sizes (2048+) and many parameters, achieving state-of-the-art image quality on ImageNet (IS ~250, FID ~7). Key techniques: class conditioning via projection into D and G via AdaIN-style modulation; truncation trick (sample z from truncated normal for quality-diversity trade-off); spectral normalization; self-attention layers for long-range dependencies. BigGAN showed that scale + careful regularization dramatically improves GAN quality, and influenced the 'scaling laws' perspective in generative modeling.

The truncation trick samples z from a truncated normal distribution (|z_i| < ψ) rather than the full normal. Smaller ψ gives sharper, more typical images (lower FID, lower IS diversity); larger ψ gives more varied but potentially lower-quality images. It allows a quality-diversity trade-off at inference time without retraining. The truncation threshold ψ=1.0 is often used as a default; ψ=0.5 for maximum quality. The trick exploits that the generator was trained on the full distribution but can be used in a subset for controlled output.

import numpy as np
z = np.random.randn(batch, latent_dim)
z = np.clip(z, -psi, psi)  # truncation

Self-Attention GAN (SAGAN) adds a self-attention layer to the generator and discriminator to capture long-range dependencies that convolutions miss. In standard CNNs, a large receptive field requires many layers; self-attention computes dependencies between all spatial positions directly: Attention(Q,K,V) = softmax(Q K^T / √d) V. This allows G to be consistent across distant image regions (e.g., consistent fur texture across a dog's body). Spectral normalization in both G and D stabilizes SAGAN training.

Signs of healthy training: D loss ≈ 0.693 (log 2, D outputs ≈ 0.5 for all), G loss decreasing, generated samples improve visually over time. Signs of trouble: D loss → 0 (D winning, G gets no gradient), D loss oscillating wildly (unstable), G produces same sample always (mode collapse), generated images have checkerboard artifacts (transposed conv stride mismatch). Monitoring generated sample grids, loss curves, and FID every N steps is essential for diagnosing and correcting GAN training issues.

Checkerboard artifacts appear when transposed convolutions (deconv) have stride > kernel size, creating uneven overlaps where some output pixels get more contributions than others. The pattern of over/under-coverage creates a grid-like artifact in generated images. Fix: replace transposed convolutions with nearest-neighbor (or bilinear) upsampling followed by a regular convolution — this gives uniform coverage. Alternatively use subpixel convolution (PixelShuffle). DCGAN authors first diagnosed this; it's now a standard knowledge point in GAN design.

# Fix: upsample then conv
x = F.interpolate(x, scale_factor=2)
x = conv(x)

Feature matching trains G to match intermediate feature statistics of D on real data, rather than directly fooling D. The loss is: ||E[f(x)] − E[f(G(z))]||², where f(x) is D's intermediate activation. This prevents G from over-optimizing D at the current step (which could lead to oscillation). Feature matching stabilizes training, especially in semi-supervised settings. It's a form of moment matching between real and generated feature distributions, connecting GANs to Maximum Mean Discrepancy (MMD) methods.

GAN inversion finds the latent code z* such that G(z*) ≈ x for a given real image x. This enables editing real images in the GAN's latent space. Methods: (1) Optimization-based: minimize ||G(z) − x||² over z via gradient descent. (2) Encoder-based: train an encoder E: x → z. (3) Hybrid: use encoder as initialization, then optimize. GAN inversion is key to semantic editing (modify z in meaningful directions to change age, expression, etc.) and is used in StyleGAN-based image editing tools.

# Optimization-based inversion
z = torch.randn(1, latent_dim, requires_grad=True)
optimizer = torch.optim.Adam([z])
for _ in range(500):
    loss = ((G(z) - x)**2).mean()
    optimizer.zero_grad(); loss.backward(); optimizer.step()

WGAN-GP improves upon WGAN by replacing weight clipping (which limits model capacity and causes slow convergence) with a gradient penalty: GP = E[(||∇_x̂ D(x̂)||_2 − 1)²], where x̂ is a linear interpolation between real and fake samples. This soft-enforces the 1-Lipschitz constraint. WGAN-GP loss: E[D(G(z))] − E[D(x)] + λ GP. WGAN-GP trains more stably, supports more complex architectures, and is the most widely used Wasserstein variant. λ=10 is a common default.

# Gradient penalty
alpha = torch.rand(bs, 1, 1, 1)
x_hat = alpha*real + (1-alpha)*fake
pred = D(x_hat)
grads = torch.autograd.grad(pred.sum(), x_hat)[0]
gp = ((grads.norm(2, dim=1) - 1)**2).mean()

An autoencoder learns to compress data to a latent code and reconstruct it — it's a deterministic mapping trained to minimize reconstruction loss (pixel-wise). The latent space may not be smooth or complete, so random sampling from it gives poor results. GANs train a generator from a structured prior (Gaussian noise) using adversarial feedback — no explicit reconstruction loss. GANs produce sharper, more realistic images because D evaluates perceptual realism, not pixel accuracy. Autoencoders are good for compression; GANs for photorealistic generation.

Flow-based models (e.g., RealNVP, Glow) learn an invertible mapping f: x → z between data space and a simple prior. Because f is invertible, exact log-likelihood p(x) = p_z(f(x)) |det J_f| can be computed and maximized. Unlike GANs, flows: (1) train with exact likelihood (no adversarial instability); (2) enable exact inference (encode and decode); (3) give a smooth, well-defined latent space. The downside: the invertibility constraint limits architecture choices and inference/generation can be slower.

Batch normalization (BN) in the generator normalizes intermediate activations to zero mean and unit variance, which stabilizes gradients and speeds training. In DCGAN, BN is applied in both G (all layers except output) and D (all layers except input). However, BN can cause issues in D for WGAN because it changes the Lipschitz property — layer normalization or no normalization is preferred for the D in WGAN-GP. BN in G generally helps; BN in D is more architecture-specific and sometimes replaced by spectral normalization or instance normalization.

Video GANs extend image GANs to generate temporally coherent video sequences. Challenges: temporal consistency (adjacent frames should be smooth), computational cost (3D convolutions or LSTM + 2D convolutions), and large data requirements. MoCoGAN (Motion and Content GAN) separates content (static scene) and motion (temporal trajectory) in the latent space: z_C (shared across frames) + z_M (sequential motion). The discriminator includes both spatial and temporal discriminators. Video GANs are used for video prediction, generation, and action simulation.

The Jensen-Shannon divergence JSD(P||Q) = 0.5 KL(P||M) + 0.5 KL(Q||M), where M = (P+Q)/2, is symmetric and bounded in [0, log 2]. Goodfellow et al. showed that the optimal GAN objective minimizes: C(G) = −log 4 + 2 JSD(p_data || p_G). So training GANs is equivalent to minimizing JSD between real and generated distributions. The problem: when p_data and p_G don't overlap (common early in training), JSD = log 2 is constant — no gradient. This motivated WGAN's switch to the Wasserstein distance, which provides useful gradients even when distributions don't overlap.

Semantic image synthesis generates photorealistic images from semantic segmentation maps (where each pixel is a class label). SPADE (Park et al. 2019) uses spatially-adaptive normalization: normalization parameters (γ, β) are learned per-pixel from the segmentation map rather than from batch statistics. This preserves spatial semantic information that standard BN destroys. SPADE achieves remarkably realistic landscape, street, and face synthesis from coarse layout maps. It extends cGANs to high-resolution, spatially-structured conditioning.

StackGAN generates images from text descriptions in two stages: Stage I generates a 64×64 rough shape conditioned on text embeddings; Stage II takes the Stage I result + text to generate a 256×256 refined image. Text is encoded (e.g., via RNN) and conditioned via Conditioning Augmentation (sample from text embedding distribution rather than directly using it). StackGAN was influential but largely superseded by diffusion-based methods (DALL-E 2, Stable Diffusion) that produce higher quality and more diverse text-to-image results.

Face aging uses conditional GANs to synthesize how a person would look at different ages. The generator takes a face image + target age label and produces an aged/de-aged version. The discriminator checks both realism and age consistency. Key challenges: preserving identity while changing age-related features (wrinkles, hair, skin texture), and requiring labeled training data with the same person at multiple ages. Face aging GANs are used in entertainment, age progression software, and missing children identification.

Supervised learning minimizes a fixed loss (e.g., cross-entropy) on labeled data — the optimization landscape is stationary. GAN training is a minimax game where the loss function changes as D and G co-evolve. There is no fixed convergence criterion; the game may oscillate. GANs have no explicit target output — D only says 'real or fake'. Training requires careful hyperparameter tuning, architecture balance, and monitoring of both G and D losses simultaneously. GANs are significantly harder to train and debug than supervised models.

Disentanglement means different dimensions of z independently control different semantic attributes (e.g., z_1 = smile, z_2 = hair color, z_3 = age). A disentangled latent space allows independent attribute editing. InfoGAN achieves disentanglement by maximizing mutual information between a latent code c and G(z,c), training a Q network to predict c from G(z,c). StyleGAN's w space is more disentangled than the input z space because the mapping network f: z→w learns a linearized representation of the attribute manifold.

Theoretically, the global optimum of a GAN is: p_G = p_data (generated distribution equals real data distribution), with D*(x) = 1/2 everywhere (D can't distinguish real from fake). At this optimum, the minimax value is V(D*,G*) = −log 4. This is a saddle point: G minimizes, D maximizes. In practice this optimum is never exactly reached — training finds a local or approximate solution. The theoretical optimum assumes infinite capacity and data, perfect optimization, and no mode collapse.

Deepfakes use GANs (or autoencoders) to replace one person's face with another in images/video. Encoder-decoder pairs are trained for each identity; the encoder is shared (learns identity-agnostic features), while decoders are identity-specific. At inference, face A's encoding is decoded with face B's decoder. GANs improve realism via adversarial refinement. This raises serious ethical issues around consent, misinformation, and fraud. Detection methods (deepfake detectors) are an active research area and an arms race with generation quality.

If D's learning rate is much higher than G's, D becomes too strong too fast — G gets near-zero gradients from a near-perfect D. If G's learning rate is too high, G overshoots and D can't keep up, causing unstable oscillations. Standard practice: use equal or slightly lower G learning rate, Adam optimizer with β_1=0.5 (less momentum to handle non-stationary loss), and sometimes train D more steps per G step (especially for WGAN). Two-timescale update rule (TTUR) uses different LRs provably to improve convergence.

# TTUR: different learning rates
optimizer_D = torch.optim.Adam(D.parameters(), lr=4e-4, betas=(0.5, 0.9))
optimizer_G = torch.optim.Adam(G.parameters(), lr=1e-4, betas=(0.5, 0.9))

Energy-Based GANs (EBGANs) model D as an energy function E(x) instead of a binary classifier — lower energy = more real. D is trained to assign low energy to real data and high energy to fake. G is trained to generate samples with low energy. The discriminator can be an autoencoder (energy = reconstruction error): real images reconstruct well (low energy); fakes don't. This gives D a richer training signal than binary classification and avoids saturation issues. EBGANs were an early alternative to standard adversarial training.

Semi-supervised GAN (e.g., Improved GAN by Salimans et al.) uses the discriminator for K+1 class classification: K real classes + 1 'fake' class. Labeled real data trains D on K classes; unlabeled real data contributes via the real/fake distinction; G generates fake samples. This gives D strong supervision from labeled data and regularizes it to be a good feature extractor. The resulting D can be used as a classifier achieving near-supervised performance with very few labels, making GANs useful for low-label-data scenarios.

The interpolation test linearly interpolates between two latent codes z_1 and z_2: z_t = (1−t)z_1 + tz_2, t ∈ [0,1], and generates images G(z_t). A good GAN produces smooth, semantically meaningful transitions (gradual face aging, style shift). Sudden jumps or discontinuities indicate a poorly structured latent space or mode collapse. Circular interpolation (slerp) is preferred for high-dimensional Gaussians because linear interpolation passes through lower-density regions. A smooth, coherent interpolation is strong evidence of good generalization.

def slerp(z1, z2, t):
    omega = np.arccos(np.dot(z1/np.linalg.norm(z1), z2/np.linalg.norm(z2)))
    return (np.sin((1-t)*omega)*z1 + np.sin(t*omega)*z2) / np.sin(omega)

GANs extend far beyond image synthesis: (1) Data augmentation — generate synthetic training data for rare classes. (2) Drug discovery — generate valid molecular structures (MolGAN). (3) Text-to-speech / voice cloning. (4) Super-resolution (SRGAN). (5) Anomaly detection — train GAN on normal data; high reconstruction error = anomaly. (6) Domain adaptation — translate data between domains. (7) Video prediction. (8) 3D object generation (3D-GAN). (9) Music generation. The adversarial principle of G vs D is broadly applicable wherever you need to model a complex distribution.

SRGAN (Ledig et al. 2016) uses a GAN to super-resolve low-resolution images to 4× higher resolution. The generator upsamples via residual blocks + sub-pixel convolution (PixelShuffle). The loss combines: (1) perceptual loss — MSE in VGG feature space (not pixel space), encouraging perceptually similar but not pixel-exact outputs; (2) adversarial loss — D pushes G toward photorealistic textures. SRGAN produces sharper, more detailed textures than PSNR-optimized methods (which tend to be blurry) at the cost of slightly higher pixel error.

Unconditional GANs generate random samples from p_data with no control over output. Class-conditional GANs (e.g., BigGAN, cGAN) condition on class label y, generating G(z,y) that looks like class y. D also receives y and checks if (x,y) is a real matching pair. This allows targeted generation ('generate a corgi') and enables class-specific FID measurement. Projection conditioning (multiply label embedding with D's final feature) outperforms concatenation. Class conditioning dramatically improves quality because the generator only needs to model one mode at a time.

GANs are implicit generative models: they define a distribution through a sampling procedure (push noise through G) without explicitly specifying the probability density p(x). This contrasts with explicit models (VAEs, flows, diffusion) that define p_θ(x) directly. Implicit models can represent complex, multimodal distributions without the curse of specifying a tractable density, but they can't compute likelihoods, making evaluation and comparison harder. The adversarial training objective sidesteps density estimation by using D as a learned density ratio estimator between p_data and p_G.