Generative adversarial networks (GANs)

The minimax objective is min_G max_D E_{x~p_data}[log D(x)] + E_{z~p_z}[log(1-D(G(z)))]. The discriminator maximizes correct classification while the generator minimizes the discriminator's success. At the global optimum the generator distribution equals the data distribution (p_g = p_data) and D(x)=1/2 everywhere, meaning the discriminator cannot distinguish real from fake.

For a fixed generator, the optimal discriminator is D*(x) = p_data(x) / (p_data(x) + p_g(x)). Substituting this back into the value function shows the generator minimizes the Jensen-Shannon divergence between p_data and p_g (up to a constant of -log 4).

The original loss min log(1-D(G(z))) saturates when the discriminator confidently rejects fakes early in training, producing near-zero gradients for the generator. The non-saturating variant instead maximizes log D(G(z)) (i.e., minimizes -log D(G(z))), which provides strong gradients precisely when the generator is performing poorly, accelerating early learning. Note it no longer optimizes the exact JS divergence.

Mode collapse occurs when the generator maps many inputs z to a few outputs, covering only a subset of the data distribution's modes while still fooling the discriminator. A common cause is the generator exploiting a discriminator that has not yet learned to penalize lack of diversity, especially under the standard objective which does not explicitly reward coverage.

WGAN replaces the JS-divergence-based objective with the Earth Mover (Wasserstein-1) distance, using a critic that outputs unbounded real scores rather than a probability. This provides smoother, more meaningful gradients even when the supports of p_data and p_g do not overlap, mitigating vanishing gradients and improving training stability.

The original WGAN enforces the 1-Lipschitz constraint via weight clipping, restricting critic weights to a fixed range [-c, c]. The constraint is required because the Kantorovich-Rubinstein duality used to compute the Wasserstein distance only holds over 1-Lipschitz functions. Weight clipping is crude and can cause capacity underuse or exploding/vanishing gradients, which motivated the gradient penalty in WGAN-GP.

WGAN-GP enforces the Lipschitz constraint by penalizing the squared deviation of the critic's gradient norm from 1, evaluated at points interpolated between real and generated samples: lambda * E[(||grad_x_hat D(x_hat)||_2 - 1)^2]. This replaces weight clipping, allowing the critic to use its full capacity while keeping gradients well-behaved.

eps = torch.rand(real.size(0), 1, 1, 1, device=real.device)
x_hat = (eps * real + (1 - eps) * fake).requires_grad_(True)
grad = torch.autograd.grad(D(x_hat).sum(), x_hat, create_graph=True)[0]
grad = grad.view(grad.size(0), -1)
gp = ((grad.norm(2, dim=1) - 1) ** 2).mean()

DCGAN replaces pooling with strided convolutions (discriminator) and fractional-strided/transposed convolutions (generator), uses batch normalization in both networks (except the generator output and discriminator input layers), removes fully connected hidden layers, uses ReLU in the generator (Tanh at output), and LeakyReLU in the discriminator. These choices stabilized convolutional GAN training and enabled meaningful latent-space arithmetic.

A conditional GAN feeds auxiliary information y (e.g., a class label) to both the generator and discriminator, typically by concatenation or embedding, so the objective becomes conditioned on y: the generator models p(x|y) and the discriminator evaluates (x, y) pairs. This enables controllable, class-targeted generation instead of sampling from the marginal data distribution.

CycleGAN introduces a cycle-consistency loss: translating an image from domain A to B and back to A should reconstruct the original (and vice versa), enforced via L1 reconstruction loss alongside two adversarial losses. This constrains the otherwise underdetermined mapping so the model can learn translations without paired training examples.

ProGAN addresses instability and slow convergence when generating high-resolution images by starting training at low resolution (e.g., 4x4) and incrementally adding layers to both generator and discriminator to reach higher resolutions (e.g., 1024x1024). New layers are faded in smoothly, which stabilizes training and speeds it up by first learning coarse structure before fine detail.

StyleGAN maps the latent z through an 8-layer MLP mapping network to an intermediate latent space W, which is less entangled than z. The w vector is injected at each layer via Adaptive Instance Normalization (AdaIN), controlling per-layer styles, while per-pixel noise adds stochastic detail. This decouples high-level attributes from stochastic variation and enables fine-grained, scale-specific style control.

StyleGAN2 replaced AdaIN with weight (de)modulation because AdaIN's instance normalization caused characteristic 'blob'/droplet artifacts: the generator created strong localized spikes to bypass the normalization. Weight demodulation scales convolution weights based on the style and normalizes by the expected output statistics, removing the droplet artifacts while preserving style control without explicit per-feature normalization.

FID embeds real and generated images using an Inception network's pooling features, models each set as a multivariate Gaussian, and computes the Fréchet (Wasserstein-2) distance between them: ||mu_r - mu_g||^2 + Tr(Sigma_r + Sigma_g - 2(Sigma_r Sigma_g)^{1/2}). Unlike the Inception Score, FID compares generated samples to real data statistics, detects mode collapse, and correlates better with human judgments of quality. Lower is better.

The Inception Score measures both sample quality (each image should have a low-entropy/confident class prediction) and diversity (the marginal over all generated images should have high entropy), computed as exp(E_x[KL(p(y|x) || p(y))]). Its main limitation is that it never compares against the real data distribution, so it cannot detect mode collapse within classes and can be gamed by adversarial or unrealistic images that classify confidently.

Spectral normalization divides each layer's weight matrix by its largest singular value (spectral norm), constraining the Lipschitz constant of the discriminator to 1. It is computationally cheap (estimated via power iteration), requires no extra hyperparameters, and stabilizes training by preventing the discriminator from producing overly sharp decision boundaries with exploding gradients.

def spectral_norm_w(W, u, n_iter=1):
    # W: (out, in), u: (out,)
    for _ in range(n_iter):
        v = F.normalize(W.t() @ u, dim=0)
        u = F.normalize(W @ v, dim=0)
    sigma = u @ W @ v          # largest singular value
    return W / sigma, u

GAN training is a non-convex two-player minimax game where the goal is a Nash equilibrium, not a single loss minimum. Simultaneous gradient descent on the two players can produce rotational/cyclic dynamics that fail to converge even on simple problems, and the equilibrium may not be reachable by gradient methods. This causes oscillation, divergence, and sensitivity to hyperparameters.

Feature matching changes the generator objective to match the expected feature statistics of an intermediate discriminator layer between real and generated data, i.e., minimize ||E_x f(x) - E_z f(G(z))||^2, rather than directly maximizing the discriminator's output. This gives the generator a more stable, less exploitable target and helps prevent overtraining on the current discriminator and mode collapse.

Minibatch discrimination lets the discriminator look at multiple examples within a minibatch jointly (e.g., by computing pairwise similarities across the batch) so it can detect when generated samples lack diversity. By penalizing batches that are too similar, it directly combats mode collapse, encouraging the generator to produce varied outputs.

TTUR assigns different learning rates to the generator and discriminator (often a higher rate for the discriminator), which under mild conditions provably allows convergence to a local Nash equilibrium. Practically, letting the discriminator learn faster keeps its estimate accurate enough to give the generator useful gradients, improving stability and final FID.

One-sided label smoothing replaces the real target of 1.0 with a smaller value like 0.9 for the discriminator on real samples, while keeping fake targets at 0. This prevents the discriminator from becoming overconfident and reduces vanishing gradients to the generator. It is one-sided because also smoothing the fake (zero) targets shifts the optimal discriminator toward p_data + (alpha)p_g in the denominator, which can reinforce poor generator samples in low-density regions.

Pix2Pix uses a PatchGAN discriminator that classifies overlapping NxN patches as real/fake rather than scoring the whole image. This focuses the adversarial loss on high-frequency local texture/structure (the L1 loss handles low-frequency correctness), reduces parameters, and runs convolutionally over arbitrarily large images, improving sharpness in conditional image translation.

A standard discriminator outputs a probability (0-1) via a sigmoid and is trained with cross-entropy to classify real vs fake. A WGAN critic outputs an unbounded real-valued score (no sigmoid) approximating the Wasserstein distance, is constrained to be 1-Lipschitz, and its output magnitude meaningfully correlates with sample quality rather than just classification confidence.

Smoothly interpolating between two latent vectors z1 and z2 and observing the generated outputs tests whether the generator has learned a meaningful, continuous manifold rather than memorizing training samples. Smooth, semantically coherent transitions indicate good generalization and disentanglement, whereas abrupt jumps or non-image outputs suggest memorization or a poorly structured latent space.

In AC-GAN, the discriminator both decides real vs fake AND predicts the class label via an auxiliary classification head, while the generator is conditioned on the class. The added classification loss encourages the generator to produce class-consistent samples and improves training, differing from cGAN where the label is merely an input to the discriminator rather than something it must reconstruct.

When p_data and p_g have negligible overlap (common in high dimensions early in training), the optimal discriminator can perfectly separate them, the JS divergence saturates at its maximum (log 2), and its gradient with respect to the generator vanishes. This leaves the generator without a useful learning signal, which is precisely the issue Wasserstein distance and the non-saturating loss aim to mitigate.

For generative models, precision measures the fraction of generated samples that fall within the support of the real data distribution (sample quality/realism), while recall measures the fraction of the real distribution covered by the generated samples (diversity/coverage). They disentangle the two failure modes that a single FID number conflates: low recall signals mode collapse, low precision signals poor-quality samples.

Path length regularization encourages that a fixed-size step in the latent space W produces a fixed-magnitude change in image space, making the generator mapping smoother and more well-conditioned. This improves image quality, makes the latent space more linear/invertible (aiding GAN inversion and editing), and correlates with better FID and PPL (perceptual path length) scores.

BigGAN scales up batch size and model width, uses class-conditional batch norm, shared embeddings, orthogonal regularization, and self-attention to achieve high-fidelity ImageNet samples. The truncation trick samples the latent z from a truncated normal at inference: smaller truncation thresholds increase sample fidelity at the cost of diversity, exposing an explicit quality-diversity trade-off knob.

GAN inversion finds the latent code (in z, W, or W+ space) that, when passed through a pretrained generator, reconstructs a given real image. It is essential for real-image editing because once an image is mapped to the latent space, semantic manipulations (e.g., adding a smile) can be performed by moving along learned latent directions. The trade-off is between reconstruction accuracy and editability, often favoring richer spaces like W+.

The hinge loss, widely used in SAGAN/BigGAN, trains the discriminator with L_D = E[max(0, 1 - D(x))] + E[max(0, 1 + D(G(z)))] and the generator with L_G = -E[D(G(z))]. It focuses the discriminator's gradient on margin-violating samples (it stops penalizing already-correctly-and-confidently classified ones), pairs well with spectral normalization, and often yields more stable training and better sample quality than the cross-entropy loss.

loss_d = F.relu(1.0 - d_real).mean() + F.relu(1.0 + d_fake).mean()
loss_g = -d_fake.mean()

Self-Attention GAN (SAGAN) inserts a self-attention layer so the generator and discriminator can model long-range, global dependencies across distant spatial regions, which purely convolutional layers (with limited receptive fields) struggle to capture. This improves generation of geometrically coherent structures (e.g., consistent object parts across the image) and notably improved ImageNet FID/IS.

Diffusion models learn to reverse a gradual noising process via a stable denoising objective (a regression/likelihood-style loss), avoiding the adversarial minimax game. They typically offer more stable training, better mode coverage/diversity, and have surpassed GANs on FID for many benchmarks, at the cost of slower iterative sampling. GANs remain competitive where single-step, low-latency generation is required, and hybrid GAN-diffusion methods (e.g., distillation/consistency models) aim to combine fast sampling with high quality.

Differentiable augmentation applies the same differentiable transformations (color, translation, cutout) to both real and generated images during training so gradients flow through them. It prevents the discriminator from overfitting/memorizing a small dataset, which otherwise causes the generator's training signal to collapse. Adaptive Discriminator Augmentation (ADA) tunes the augmentation probability automatically based on discriminator overfitting signals, enabling high-quality GANs with only a few thousand images.