Autoencoders

An autoencoder is a type of neural network trained to reconstruct its own input at the output. It consists of an encoder that compresses the input into a lower-dimensional code and a decoder that rebuilds the input from that code. By forcing data through a bottleneck, it learns a compact representation of the data without needing labels.

An autoencoder has an encoder and a decoder. The encoder maps the input to a compressed latent representation (the code), and the decoder maps that code back to a reconstruction of the original input. They are trained together to minimize reconstruction error.

The bottleneck is the middle layer with fewer units than the input, where the data is represented in compressed form (the latent code). It forces the network to keep only the most important features needed to reconstruct the input. This compression is what makes the autoencoder learn useful representations.

Autoencoders do not need human-provided labels; the target output is simply the input itself. The network supervises itself by trying to reproduce its input, which is why this is often called self-supervised learning. This lets them learn from large amounts of unlabeled data.

Mean Squared Error (MSE) is commonly used, measuring the average squared difference between the input and its reconstruction. Minimizing MSE makes the output as close as possible to the original input. For binary or normalized pixel-intensity data, binary cross-entropy is another common choice.

An undercomplete autoencoder has a latent layer smaller than the input, forcing compression and feature learning. An overcomplete autoencoder has a latent layer equal to or larger than the input, which risks simply copying the input unless regularization is applied. Undercomplete designs naturally prevent trivial identity learning.

If the latent layer is large enough (overcomplete) with no constraints, the network can learn the trivial identity function and copy the input without learning useful features. This is prevented by using a bottleneck (undercomplete design), adding regularization, or injecting noise (denoising autoencoders). These constraints force the network to capture meaningful structure.

A denoising autoencoder is trained by feeding corrupted (noisy) versions of the input while asking it to reconstruct the clean original. This forces the network to learn robust features that capture the underlying structure rather than just memorizing pixels. It is useful for noise removal and learning more meaningful representations.

The encoder maps high-dimensional input to a low-dimensional latent code, which serves as a compressed representation. This code can be used in place of the original data for visualization or as input to other models. Unlike PCA, autoencoders can capture nonlinear relationships in the data.

PCA performs only linear dimensionality reduction, finding directions of maximum variance. An autoencoder with nonlinear activation functions can learn nonlinear, more flexible compressions of the data. In fact, a linear autoencoder with MSE loss can recover the same subspace as PCA.

Reconstruction error measures how different the autoencoder's output is from the original input, often computed as MSE. It is the quantity minimized during training and indicates how well the latent code preserves information. High reconstruction error on new data can also signal that the input is unusual (useful for anomaly detection).

An autoencoder trained mostly on normal data learns to reconstruct normal patterns with low error. When given an anomalous input it has never seen, it reconstructs it poorly, producing a high reconstruction error. By setting a threshold on this error, anomalies can be flagged automatically.

A VAE is a generative autoencoder that encodes inputs into a probability distribution (typically a mean and variance) rather than a single point. By sampling from this latent distribution and decoding, it can generate new data similar to the training set. It adds a KL-divergence term to keep the latent space smooth and continuous.

The KL-divergence term pushes the learned latent distributions to stay close to a standard normal distribution. This regularizes the latent space so it is smooth and continuous, allowing meaningful sampling and interpolation. The total VAE loss balances this term against the reconstruction loss.

Sampling from a distribution is not differentiable, which blocks backpropagation. The reparameterization trick rewrites a sample as mean + standard_deviation * epsilon, where epsilon is random noise from a standard normal. This moves the randomness outside the network path so gradients can flow through the mean and variance.

A sparse autoencoder adds a penalty (such as an L1 or KL sparsity term) encouraging most latent neurons to be inactive for any given input. This forces the network to use only a few units to represent each example, learning more interpretable, feature-specific representations. Sparsity can be applied even when the latent layer is large.

The encoder is a function (a set of neural network layers) that maps the input x to a latent code z, often written z = f(x). It progressively reduces dimensionality through weighted layers and activations. The resulting code z is a compressed summary of the input.

The decoder is a function that maps the latent code z back to a reconstruction of the input, often written x_hat = g(z). It expands the compressed code through layers and activations to approximate the original. Training adjusts both encoder and decoder so x_hat is close to x.

Without nonlinear activations, stacking layers would still only produce a linear transformation, making the autoencoder no more powerful than PCA. Nonlinearities like ReLU or sigmoid let the network model complex, curved relationships in the data. This enables it to learn richer, more useful compressed representations.

A convolutional autoencoder uses convolutional layers in the encoder and transposed/upsampling layers in the decoder. It is well suited to image data because convolutions capture spatial patterns efficiently with shared weights. This makes it more parameter-efficient and effective than fully connected autoencoders for images.

A vanilla autoencoder is not designed for generation because its latent space can have gaps and is not organized as a smooth probability distribution. Randomly chosen latent points often decode to meaningless outputs. Variational autoencoders solve this by structuring the latent space, making sampling meaningful.

The VAE loss combines a reconstruction term (how well the output matches the input) and a KL-divergence regularization term (how close the latent distribution is to a standard normal). The reconstruction term encourages accurate outputs while the KL term shapes the latent space. Balancing the two yields a useful generative model.

If the bottleneck is too small, the latent code cannot hold enough information to reconstruct the input accurately, leading to high reconstruction error and loss of important details. The model underfits because it is over-compressed. Choosing the latent size is a trade-off between compression and reconstruction quality.

A contractive autoencoder adds a penalty on the Jacobian (the derivatives of the latent code with respect to the input). This encourages the learned representation to change little when the input is slightly perturbed, making features robust to small variations. It learns representations that are stable along directions where the data does not vary.

The encoder's job (producing a good code) and the decoder's job (reconstructing from that code) are interdependent, so they must learn compatible mappings. Training them jointly with a single reconstruction loss lets gradients flow through both, aligning the code with what the decoder can use. Separate training would not coordinate the shared latent representation.

After training, the encoder can be reused as a feature extractor, turning raw inputs into compact latent vectors. These vectors often capture meaningful structure and can feed a simpler classifier or clustering algorithm, even with limited labels. This leverages unsupervised pretraining to improve downstream performance.

The input is a noisy (corrupted) version of the image, while the target it must reconstruct is the original clean image. The network learns to map noisy inputs to clean outputs, effectively learning to remove noise. At test time it can denoise new corrupted images.

A smooth latent space means nearby points decode to similar, meaningful outputs, with no large gaps of nonsense. This allows interpolation between examples and meaningful sampling for generation. VAEs explicitly encourage smoothness through the KL regularization term.

A simple autoencoder stacks a Dense encoder layer that reduces dimensionality, then a Dense decoder layer that restores the original size, compiled with a reconstruction loss. The encoding_dim must be smaller than the input dimension to create a bottleneck. The model is trained with the input as both x and y.

from tensorflow.keras import layers, Model, Input
inp = Input(shape=(784,))
code = layers.Dense(32, activation='relu')(inp)   # bottleneck
out = layers.Dense(784, activation='sigmoid')(code)
ae = Model(inp, out)
ae.compile(optimizer='adam', loss='mse')
ae.fit(x_train, x_train, epochs=10)  # target = input

An autoencoder can overfit by memorizing training examples instead of learning general structure, especially with a large latent layer or small dataset. This leads to poor reconstruction of unseen data. Techniques like adding noise (denoising), sparsity penalties, weight regularization, or smaller bottlenecks help reduce overfitting.

A linear autoencoder (no nonlinear activations) trained with MSE loss learns to project data onto the same subspace as PCA. It captures the directions of greatest variance, though it may not produce the exact orthogonal principal components without extra constraints. This shows PCA is essentially a special, linear case of autoencoding.

A stacked autoencoder has multiple hidden layers in both the encoder and decoder, giving it depth. The extra layers let it learn hierarchical, increasingly abstract features and more complex compressions than a shallow autoencoder. It can be trained end-to-end or, historically, layer by layer.

The target is the input example itself; the network tries to make its output equal the input. This is why no separate labels are required and why it is self-supervised. The loss compares the reconstruction to this input target.

Because autoencoders learn from unlabeled data, they can extract useful features from the large pool of raw examples. The learned encoder representations can then be combined with the few available labels to train a more accurate downstream model. This makes autoencoders valuable for semi-supervised settings.