Introduction to deep learning

Deep learning is a subfield of machine learning that uses artificial neural networks with many layers (hence 'deep') to automatically learn hierarchical representations from data. Unlike traditional machine learning, which often relies on hand-engineered features, deep learning learns features directly from raw data. It is especially powerful for tasks like image recognition, speech, and natural language processing.

An artificial neuron computes a weighted sum of its inputs, adds a bias, and passes the result through an activation function to produce an output. Mathematically, output = activation(w1*x1 + w2*x2 + ... + b). It is the basic building block of neural networks, loosely inspired by biological neurons.

Activation functions introduce non-linearity, allowing the network to model complex relationships. Without them, stacking multiple layers would still only compute a linear function, no matter how many layers exist. Non-linear activations like ReLU or sigmoid let deep networks approximate complicated functions.

ReLU (Rectified Linear Unit) is defined as f(x) = max(0, x), outputting the input if positive and 0 otherwise. It is popular because it is computationally simple and helps mitigate the vanishing gradient problem, allowing deeper networks to train faster. However, it can suffer from 'dying ReLU' where neurons output zero permanently.

import numpy as np
def relu(x):
    return np.maximum(0, x)
print(relu(np.array([-2, 0, 3])))  # [0 0 3]

The sigmoid function is f(x) = 1/(1 + e^(-x)), squashing any input into the range (0, 1), which makes it useful for representing probabilities. A major drawback is the vanishing gradient problem: for very large or very small inputs, the gradient becomes nearly zero, slowing learning in deep networks.

A feedforward neural network is one in which information flows in a single direction, from the input layer through hidden layers to the output layer, with no cycles or loops. Each layer's outputs become the next layer's inputs. It is the simplest type of artificial neural network architecture.

Weights determine the strength of the connection between neurons, scaling how much each input influences the output. The bias shifts the activation function, allowing the neuron to fit data that does not pass through the origin. Both are learnable parameters adjusted during training to minimize error.

A loss function measures how far the model's predictions are from the true target values, producing a single number that quantifies error. Training aims to minimize this loss by adjusting weights and biases. Common examples include mean squared error for regression and cross-entropy for classification.

Gradient descent is an optimization algorithm that iteratively adjusts parameters in the direction that most reduces the loss, using the negative gradient of the loss function. Each step updates weights as w := w - learning_rate * gradient. Repeating this gradually moves the model toward a minimum of the loss.

w = w - learning_rate * gradient

The learning rate controls the size of the steps taken during gradient descent when updating weights. If it is too large, training may overshoot and diverge; if too small, training becomes very slow and may get stuck. Choosing a good learning rate is crucial for effective and stable training.

Backpropagation is the algorithm used to compute the gradient of the loss with respect to each weight by applying the chain rule of calculus, working backward from the output layer to the input layer. These gradients are then used by an optimizer like gradient descent to update the weights. It makes training deep networks efficient.

An epoch is one complete pass of the entire training dataset through the network during training. Models are usually trained for many epochs so they can gradually improve. Too few epochs can underfit, while too many can lead to overfitting.

A batch is a subset of the training data processed together before the model updates its weights once. Mini-batch training balances the stability of using all data with the speed and noise of using single samples, making training efficient and well-suited to GPU hardware. It is the most common training approach in practice.

Overfitting occurs when a model learns the training data too well, including its noise and random fluctuations, so it performs poorly on new, unseen data. It is characterized by low training error but high validation/test error. Techniques like regularization, dropout, and more data help reduce it.

Underfitting happens when a model is too simple to capture the underlying patterns in the data, resulting in high error on both training and test sets. It is often caused by insufficient model capacity, too few training epochs, or overly strong regularization. The fix is usually a more expressive model or longer training.

Dropout is a regularization technique that randomly 'drops' (sets to zero) a fraction of neurons during each training step. This prevents neurons from co-adapting too much and forces the network to learn more robust, redundant features, reducing overfitting. At test time, all neurons are used.

The training set is used to fit the model's parameters, the validation set is used to tune hyperparameters and monitor performance during development, and the test set is used once at the end to estimate real-world performance on unseen data. Keeping them separate ensures an honest evaluation. The test set must not influence training decisions.

A CNN is a type of neural network designed to process grid-like data such as images, using convolutional layers that apply learnable filters to detect local patterns like edges and textures. They are the standard architecture for computer vision tasks like image classification and object detection. Convolution makes them efficient by sharing weights across the image.

A convolutional filter is a small matrix of learnable weights that slides across the input image, computing dot products to produce a feature map that highlights specific patterns like edges or corners. Early layers detect simple features and deeper layers combine them into complex ones. Filters are shared across the whole image, greatly reducing parameters.

A pooling layer reduces the spatial dimensions of feature maps, summarizing regions to make representations smaller and more manageable. Max pooling, for instance, keeps only the largest value in each region. This reduces computation, controls overfitting, and gives some translation invariance.

An RNN is a neural network designed for sequential data, where connections form loops so that information from previous steps is carried forward via a hidden state. This makes it suitable for time series, text, and speech where order matters. However, basic RNNs struggle with long-term dependencies due to vanishing gradients.

LSTM (Long Short-Term Memory) networks address the vanishing gradient problem in standard RNNs, which makes it hard to learn long-term dependencies. They use gates (input, forget, output) and a cell state to control what information is kept or discarded over time. This lets them remember relevant information across longer sequences.

The vanishing gradient problem occurs when gradients become extremely small as they are propagated backward through many layers, causing the early layers to learn very slowly or not at all. It is common with saturating activations like sigmoid in deep networks. Solutions include ReLU activations, careful initialization, batch normalization, and architectures like LSTMs or residual connections.

Softmax converts a vector of real-valued scores into a probability distribution where all values are positive and sum to 1. It is typically used in the output layer of a multi-class classification network so the outputs can be interpreted as class probabilities. The class with the highest probability is chosen as the prediction.

import numpy as np
def softmax(x):
    e = np.exp(x - np.max(x))
    return e / e.sum()
print(softmax(np.array([2.0, 1.0, 0.1])))

Cross-entropy measures the difference between the predicted probability distribution and the true distribution (usually a one-hot label), heavily penalizing confident wrong predictions. It pairs naturally with softmax outputs and produces well-behaved gradients for classification. This typically leads to faster and more stable training than mean squared error for classification tasks.

Parameters are the internal values the model learns from data during training, such as weights and biases. Hyperparameters are settings chosen before training that control the learning process, such as learning rate, batch size, number of layers, and number of epochs. Hyperparameters are tuned manually or via search, not learned by gradient descent.

The universal approximation theorem states that a neural network with even a single hidden layer containing enough neurons can approximate any continuous function on a bounded domain to arbitrary accuracy. It explains why neural networks are so flexible. However, it does not guarantee that such a network is easy to train or that it will generalize well.

Good weight initialization keeps the scale of activations and gradients reasonable across layers, helping avoid vanishing or exploding gradients early in training. If all weights start at the same value, neurons learn identical features (the symmetry problem). Methods like Xavier/Glorot and He initialization set sensible random starting scales based on layer sizes.

Batch normalization normalizes the inputs to a layer for each mini-batch, keeping them at a stable mean and variance. This stabilizes and speeds up training, allows higher learning rates, and provides a mild regularizing effect. It helps deep networks train more reliably.

In a deep network, earlier layers learn simple, low-level features (like edges in an image), and later layers combine these into increasingly complex, high-level features (like shapes, then objects). This hierarchy is learned automatically from data rather than designed by hand. It is a key reason deep learning excels at perception tasks.

Transfer learning takes a model pre-trained on a large dataset and reuses its learned features for a new, related task, often by fine-tuning only some layers. It is useful because it requires far less data and training time than training from scratch. It is widely used in computer vision and natural language processing.

GPUs (graphics processing units) have thousands of cores that can perform many calculations in parallel, which suits the large matrix and tensor operations at the heart of neural networks. This makes training dramatically faster than on a typical CPU. As a result, GPUs (and specialized hardware like TPUs) are standard for deep learning.

A tensor is a multi-dimensional array of numbers, generalizing scalars (0-D), vectors (1-D), and matrices (2-D) to higher dimensions. Deep learning frameworks like TensorFlow and PyTorch represent data, weights, and activations as tensors. Operations on tensors can run efficiently on GPUs.

import numpy as np
# A 2x3 tensor (matrix)
t = np.array([[1, 2, 3], [4, 5, 6]])
print(t.shape)  # (2, 3)

A shallow network has only one or very few hidden layers, while a deep network has many hidden layers stacked together. Depth lets the network learn more abstract, hierarchical representations and often model complex functions more efficiently. However, deeper networks are generally harder to train and require more data.

Adam is an adaptive optimization algorithm that adjusts the learning rate for each parameter individually using estimates of the first and second moments (mean and variance) of the gradients. Compared to plain gradient descent with a fixed learning rate, Adam often converges faster and is less sensitive to the initial learning rate. It is one of the most widely used optimizers.

The exploding gradient problem occurs when gradients grow very large during backpropagation, causing unstable updates and divergence, often in deep or recurrent networks. A common remedy is gradient clipping, which caps the gradient's magnitude to a maximum value. Careful initialization and normalization also help.

In a fully connected (dense) layer, every neuron in the first layer connects to every neuron in the next, giving m * n weights. Adding biases would contribute an extra n parameters. Because the count is the product of the two layer sizes, fully connected layers can become parameter-heavy when the layers are large.

Data augmentation artificially expands a training dataset by applying label-preserving transformations, such as rotating, flipping, cropping, or adjusting the brightness of images. This increases data diversity, helps the model generalize better, and reduces overfitting. It is especially valuable when labeled data is limited.

Supervised learning means training a model on input-output pairs where the correct labels are known, and image classification with labeled images is a classic example. The deep network learns to map images to their labels by minimizing prediction error. This contrasts with unsupervised learning, where no labels are provided.

Adding layers increases model capacity but can make training harder due to issues like vanishing/exploding gradients, overfitting, and higher computational cost. Beyond a point, extra depth may hurt generalization or fail to train without techniques like residual connections and normalization. More depth helps only if there is enough data and proper training methods.