How neural networks train

Learning means the network slowly adjusts its internal numbers (called weights) so its predictions get closer to the correct answers shown in the training examples. At the start the weights are random and the guesses are bad, but with each example the network nudges them in a better direction. Over many examples, it gets good at the task without anyone writing exact rules for it.

A weight is a number that controls how strongly one neuron's output affects the next neuron. Bigger weights make a connection more important, and the network learns by changing these numbers. Training is basically the search for a good set of weights.

A loss function is a formula that measures how wrong the network's prediction is compared to the correct answer. A high loss means a bad prediction and a low loss means a good one. Training works by changing the weights to make the loss as small as possible.

Gradient descent is the method a network uses to lower its loss step by step. The gradient tells which direction increases the error, so the network moves the weights in the opposite direction to reduce it. It is like walking downhill to reach the lowest point of a valley.

The learning rate decides how big each step is when the network updates its weights. If it is too large, the network can overshoot and never settle; if it is too small, training becomes very slow. Choosing a good learning rate helps the network learn quickly and steadily.

A forward pass is when input data travels through the network from the first layer to the last to produce a prediction. Each layer transforms the numbers using its weights and an activation function. The output of the forward pass is then compared to the correct answer to measure the loss.

Backpropagation is the process of sending the error backward through the network to find out how much each weight contributed to the mistake. It uses these clues to decide how to change each weight to reduce the loss. It is the engine that makes gradient descent practical in deep networks.

An activation function adds non-linearity, which lets the network learn complex, curved patterns instead of only straight-line relationships. Without it, stacking many layers would behave like a single simple layer. Examples include ReLU and sigmoid.

An epoch is one full pass through the entire training dataset. Networks usually need many epochs so they can see the examples repeatedly and keep improving. After each epoch the weights are a little better tuned.

A batch is a small group of training examples processed together before the weights are updated once. Using batches is faster and more stable than updating after every single example. The batch size is a setting chosen before training.

Overfitting happens when a network memorizes the training examples too closely, including their noise, instead of learning general patterns. It then does well on training data but poorly on new, unseen data. The goal of training is to generalize, not just memorize.

Underfitting is when a network is too simple or undertrained, so it fails to capture the patterns even in the training data. Both training and test accuracy stay low. It can often be fixed with a bigger model, better features, or more training.

We train the network on the training set and then check it on a separate test set it has never seen. This tells us whether the network truly learned general patterns or just memorized the training examples. Without a test set we could be fooled by overfitting.

A bias is an extra number added to a neuron's input that lets it shift its output up or down, independent of the inputs. It gives the network more flexibility to fit patterns, like moving a line away from the origin. Biases are learned during training just like weights.

Random starting weights break symmetry so that different neurons learn different things instead of all becoming identical. If every weight started the same, the neurons would update identically and the network couldn't learn rich features. Random initialization gives training a useful starting point.

Convergence means the loss has stopped decreasing much and the weights have settled into a good, stable solution. Further training brings little or no improvement. At this point the model is usually ready to be evaluated or used.

A label is the correct answer attached to a training example, such as the word "cat" on a cat photo. The network compares its prediction to the label to compute the loss and learn. Supervised learning needs many labeled examples to train.

Each weight is changed by subtracting the learning rate multiplied by the gradient of the loss for that weight. The gradient points toward higher error, so subtracting it lowers the error. Repeating this rule over many steps gradually improves the network.

# Basic gradient descent update for one weight
learning_rate = 0.01
weight = weight - learning_rate * gradient

In very deep networks, gradients can shrink toward zero as they travel backward through many layers, so the early layers learn extremely slowly or not at all. This makes training deep networks hard. Techniques like the ReLU activation and special architectures help reduce it.

A validation set is data kept aside to check the model's progress while tuning settings like the learning rate or number of layers. It helps detect overfitting early and pick the best version of the model. It is separate from both the training set and the final test set.

Parameters, such as weights and biases, are the values the network learns automatically during training. Hyperparameters, such as the learning rate, batch size, and number of layers, are chosen by a human before training begins. Good hyperparameters make learning the parameters easier.

More varied examples help the network see many situations, so it learns general patterns instead of memorizing a few cases. This usually improves performance on new data and reduces overfitting. However, the data must also be good quality and relevant.

An iteration is one weight update, which happens after processing one batch. An epoch is one complete pass over all the training data, which contains many iterations. For example, 1000 examples in batches of 100 give 10 iterations per epoch.

If the training data mostly represents one group or contains unfair patterns, the network learns those biases and repeats them in its predictions. For example, a model trained only on one kind of face may work poorly on others. Using balanced, representative data helps make AI fairer.

Training accuracy is how often the network gets the training examples right. It can be misleading because a model might memorize the training data and still fail on new data. That is why we also measure accuracy on a separate test set.

Regularization is a set of techniques that discourage a model from becoming too complex, which helps prevent overfitting. A common method adds a penalty for very large weights so the model stays simpler. The result usually generalizes better to new data.

Dropout randomly turns off some neurons during each training step so the network does not rely too heavily on any single one. This forces it to learn more robust, spread-out features and helps prevent overfitting. During actual use, all neurons are active again.

Each step makes only a small adjustment to the weights because large jumps could overshoot the best solution. By taking many small steps, the network gradually reduces its error and finds good weights. This careful, repeated process is what makes learning stable.

Scaling input features to a similar range, such as 0 to 1, helps training run faster and more stably. If some inputs are huge and others tiny, the network struggles to balance their effects. Normalization keeps the gradients well-behaved and speeds up learning.

Early stopping means halting training once the validation performance stops improving, even if training accuracy could still rise. This prevents the model from overfitting by training too long. It saves time and often produces a better, more general model.

In one step the network makes a prediction (forward pass), measures how wrong it is with the loss, computes how to fix each weight (backpropagation), and then updates the weights a little. Repeating this loop many times is how the network learns. Each step lowers the error a bit.

Testing on new, unseen data shows how the model will really perform in the world rather than on memorized examples. A model that only does well on its training data could fail badly with real users. This honest check helps avoid harmful mistakes when the AI is used by people.

An optimizer is the rule that decides exactly how to use the gradients to update the weights each step. SGD takes simple steps in the downhill direction, while Adam adapts the step size for each weight to learn faster. Choosing a good optimizer can make training quicker and more stable.

This pattern is a classic sign of overfitting: the model is memorizing the training data while getting worse on new data. The growing gap shows it is no longer learning general patterns. A common fix is to stop early or add regularization.