Convolutional neural networks (CNNs)

The core operation is convolution, where a small grid of weights called a filter (or kernel) slides across the input image and computes a weighted sum (dot product) at each position. This produces a feature map that highlights where a particular pattern, such as an edge, appears in the image. Stacking many such filters lets the network detect increasingly complex patterns.

CNNs use local connectivity and weight sharing, so the same small filter is reused across the whole image instead of having a separate weight for every pixel-to-neuron connection. This dramatically reduces the number of parameters and makes the network efficient and less prone to overfitting. It also gives CNNs translation invariance, meaning a feature can be detected no matter where it appears.

A feature map is the output produced when one filter is convolved across the entire input, with one value at each spatial location indicating how strongly the filter's pattern is present there. Each convolutional layer typically produces many feature maps, one per filter. Early feature maps capture simple features like edges, while deeper ones capture complex shapes and objects.

Pooling downsamples each feature map by summarizing small regions, for example taking the maximum value in each 2x2 window (max pooling). This reduces the spatial size, lowering computation and the number of parameters, while retaining the strongest features. It also adds a degree of robustness to small shifts and distortions in the input.

The output size is given by (W - F)/S + 1 = (32 - 5)/1 + 1 = 28, so the output is 28x28. With no padding, the filter cannot extend past the borders, so the spatial dimensions shrink. This is why convolution without padding gradually reduces the size of feature maps through the network.

# (W - F)/S + 1
output = (32 - 5)//1 + 1  # = 28

Padding adds extra pixels (usually zeros) around the border of the input before convolution. 'Same' padding chooses the amount of padding so that the output feature map has the same spatial dimensions as the input (for stride 1). This prevents the rapid shrinking of feature maps and helps preserve information at the image edges.

Stride is the number of pixels the filter moves each step as it slides across the input. A larger stride makes the filter skip positions, producing a smaller output feature map and reducing computation. However, increasing stride too much can lose fine spatial detail.

ReLU (Rectified Linear Unit) outputs max(0, x), introducing non-linearity so the network can learn complex patterns beyond linear combinations. It is computationally cheap and helps reduce the vanishing gradient problem because its gradient is 1 for positive inputs. This makes deep CNNs faster and more stable to train than with sigmoid or tanh.

def relu(x):
    return max(0, x)

Translation invariance means the network can recognize a feature regardless of where it appears in the image. Weight sharing in convolution detects the same pattern at every location, and pooling further makes the response robust to small positional shifts. Together they let a CNN identify, say, a cat whether it is in the top-left or bottom-right of the image.

For a multi-channel input, each filter also has matching depth, so a 3x3 filter on an RGB image is actually 3x3x3. It computes a dot product across all three channels at each location and sums the results into a single value in the output feature map. Thus the filter's depth always equals the number of input channels.

The filter size (e.g., 3x3) is the spatial extent of each kernel's receptive field, while the number of filters is how many different kernels the layer learns. The number of filters determines the depth (number of channels) of the output feature maps. So size controls what region each filter sees, and count controls how many distinct features are detected.

Each filter has 3x3x3 = 27 weights plus 1 bias = 28 parameters. With 16 filters, the total is 16 x 28 = 448 parameters. The formula is (filter_height x filter_width x input_channels + 1) x number_of_filters.

params = (3*3*3 + 1) * 16  # = 448

The receptive field is the region of the original input image that influences a particular neuron's activation. As you stack more convolution and pooling layers, the receptive field grows, so deeper neurons 'see' larger portions of the image. This is why deep layers can detect large, complex structures while shallow layers detect small, local features.

After convolution and pooling extract spatial features, the feature maps are flattened and passed to fully connected layers that combine these high-level features to make the final decision. The last dense layer usually has one neuron per class, often followed by a softmax to produce class probabilities. They act as the classifier on top of the learned features.

Softmax converts the raw output scores (logits) into a probability distribution where all values are between 0 and 1 and sum to 1. This lets the network express confidence across multiple mutually exclusive classes. The class with the highest probability is chosen as the prediction.

Early (shallow) layers learn low-level features such as edges, colors, and simple textures. Middle layers combine these into shapes and parts like corners or eyes, and deep layers represent whole objects or high-level concepts. This automatic, hierarchical feature learning is a key advantage of CNNs over hand-engineered features.

Data augmentation creates new training samples by applying label-preserving transformations such as flipping, rotating, cropping, or adjusting brightness of existing images. This artificially enlarges and diversifies the dataset, helping the CNN generalize better and reducing overfitting. It teaches the model to be robust to variations it may encounter in real data.

Transfer learning reuses a CNN that was already trained on a large dataset (like ImageNet) and adapts it to a new, often smaller, task. Typically the early feature-extracting layers are kept (frozen) and only the final layers are retrained for the new classes. This saves time and data because the general visual features are already learned.

Batch normalization standardizes the inputs to a layer for each mini-batch, keeping their distribution more stable during training. This is often described as reducing internal covariate shift; in practice it allows higher learning rates, speeds up convergence, and adds a mild regularizing effect. As a result, deep CNNs become easier and faster to train.

Dropout randomly sets a fraction of neuron activations to zero during each training step, forcing the network not to rely too heavily on any single neuron. This reduces overfitting by encouraging redundant, robust representations. It is most commonly applied in the fully connected layers near the end of the network.

In very deep networks, gradients can shrink toward zero as they backpropagate through many layers, making early layers learn extremely slowly. Residual connections add a shortcut that lets the input bypass one or more layers and be added to their output, so gradients can flow more directly backward. This allows training of much deeper CNNs without performance degradation.

Max pooling takes the maximum value in each window, emphasizing the strongest activation (most prominent feature). Average pooling takes the mean of the window, producing a smoother, more averaged summary. Max pooling is more common in feature extraction, while average pooling (especially global average pooling) is often used near the output.

Flattening reshapes the multi-dimensional feature maps (height x width x channels) into a single one-dimensional vector. This vector can then be fed into fully connected layers, which expect a flat list of inputs. It bridges the spatial feature extractor and the final classifier.

# e.g. (7,7,64) -> 7*7*64 = 3136 values
flat_size = 7 * 7 * 64

Two stacked 3x3 convolutions have the same effective receptive field (5x5) as one 5x5 convolution but use fewer parameters (2x9=18 vs 25 weights per channel) and include an extra non-linearity between them. The added non-linearity increases the network's expressive power. This is why architectures like VGG favor small stacked filters.

Categorical cross-entropy (also called softmax cross-entropy) is the standard loss for multi-class classification. It measures the difference between the predicted probability distribution and the true one-hot label, penalizing confident wrong predictions heavily. Minimizing it pushes the network to assign high probability to the correct class.

Pooling with a 2x2 window and stride 2 halves each spatial dimension, so 28x28 becomes 14x14. The general formula is (W - F)/S + 1 = (28 - 2)/2 + 1 = 14. The depth (number of channels) stays unchanged.

out = (28 - 2)//2 + 1  # = 14

Global average pooling takes the average of each entire feature map, reducing each channel to a single number, so a HxWxC tensor becomes a vector of length C. It drastically cuts parameters compared to large dense layers and reduces overfitting. It is often used right before the final classification layer in modern CNNs.

The depth of a layer's output equals the number of filters used in that layer, since each filter produces one feature map. For example, 64 filters yield an output with depth 64. Depth is independent of the spatial height and width of the feature maps.

In PyTorch, nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, padding=1) creates this layer; padding=1 keeps the spatial size the same for a 3x3 kernel with stride 1. The in_channels must match the input's channel count, and out_channels sets the number of filters. This layer would be followed by an activation like ReLU.

import torch.nn as nn
conv = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, padding=1)

As pooling and strided convolutions shrink the spatial dimensions, the network increases the number of filters to capture a richer set of higher-level features. This trades spatial resolution for representational depth, letting deep layers encode many complex patterns compactly. It keeps the computational cost balanced across the network.

A 1x1 convolution applies a filter that covers a single spatial position but spans all input channels, effectively mixing channel information at each pixel. It is commonly used to reduce or increase the number of channels (dimensionality change) cheaply and to add non-linearity without changing spatial size. Architectures like Inception and ResNet use it to control computational cost.

Because a filter is applied at many positions, the gradient of the loss with respect to that filter is the sum of the gradients contributed at every position where it was used. This summed gradient is then used to update the single shared set of weights. Weight sharing thus means one update reflects the filter's behavior across the whole image.

An edge-detecting filter (like a Sobel kernel) has positive weights on one side and negative weights on the other, for example [[-1,0,1],[-2,0,2],[-1,0,1]] for vertical edges. When placed over a region with a sharp intensity change, the positive and negative sides produce a large summed response, while uniform regions cancel out to near zero. CNNs can learn such filters automatically from data.

# Vertical Sobel filter
kernel = [[-1,0,1],
          [-2,0,2],
          [-1,0,1]]

Overfitting occurs when a CNN learns the training data too well, including its noise, and fails to generalize to new images, shown by high training accuracy but low validation accuracy. Two common remedies are data augmentation (creating varied training images) and dropout (randomly disabling neurons during training). Other options include weight regularization and early stopping.