Math for ML

Machine learning is built on math. Statistics helps us understand data patterns. Linear algebra (vectors and matrices) is how data and neural networks are represented. Calculus is used to train models by finding the best settings. Probability lets us handle uncertainty. You don't need to be a math genius to use AI, but understanding these foundations helps you know why algorithms work — and what to do when they don't.

The mean (average) is calculated by adding all values and dividing by the count. If five students scored 60, 70, 75, 80, and 90 on a test, the mean is (60+70+75+80+90) ÷ 5 = 75. The mean is the central value in a dataset. Data scientists use it constantly — to summarize data, compare groups, and measure typical outcomes. It's sensitive to outliers: one student scoring 200 would pull the mean far up.

The median is the middle value when data is sorted. If salaries in a company are $30k, $40k, $50k, $60k, and $1,000,000, the mean is about $236,000 (misleadingly high) but the median is $50,000 (a much more representative 'typical' salary). Whenever data has extreme outliers — like income, house prices, or medical costs — the median is more useful than the mean because it isn't skewed by extreme values.

The mode is the value that appears most often in a dataset. In {3, 5, 3, 7, 8, 3}, the mode is 3. Mode is most useful for categorical data — like finding the most popular shoe size, most common eye color in a class, or most frequently ordered menu item. A dataset can have two modes (bimodal) or more. Unlike mean and median, mode can apply to non-numerical data too.

Variance measures how spread out the values in a dataset are around the mean. High variance means values are all over the place; low variance means they're tightly clustered. If all students in a class score exactly 75, variance is 0. If scores range from 20 to 100, variance is high. Machine learning uses variance to understand data spread and in model evaluation — high model variance means the model is too sensitive to small changes in training data.

Standard deviation (SD) is the square root of variance — it measures spread in the same units as the original data. If you measure heights in centimeters, SD is also in centimeters, which makes it easier to interpret than variance. A small SD means heights are close together (a very uniform class). A large SD means heights vary widely. In ML, SD is used in data normalization to scale features to comparable ranges.

Probability is a number between 0 and 1 that measures how likely an event is. 0 means impossible; 1 means certain; 0.5 means 50-50. ML models often output probabilities: 'There is a 92% probability this email is spam.' These probabilities guide decisions — the model classifies the email as spam if the probability is above a threshold (say, 0.5). Probability is also used in Bayesian methods, decision trees, and model uncertainty estimates.

A vector is an ordered list of numbers. In ML, data is almost always represented as vectors. A house might be described as the vector [3, 1500, 2, 1990] meaning 3 bedrooms, 1500 sq ft, 2 bathrooms, built in 1990. Neural networks process vectors through layers of math. Word embeddings represent words as high-dimensional vectors. Vectors let computers handle complex, multi-feature data as clean mathematical objects.

A matrix is a 2D grid of numbers arranged in rows and columns. A dataset with 1,000 training examples and 10 features is naturally represented as a 1,000×10 matrix. Neural networks store their weights in matrices. The core computation in a neural network is matrix multiplication — an incredibly fast operation on modern GPUs. Understanding matrices is essential for understanding how any ML algorithm actually works mathematically.

The dot product of two vectors multiplies corresponding numbers and adds the results. For [1,2,3] and [4,5,6]: (1×4)+(2×5)+(3×6) = 4+10+18 = 32. In ML, dot products measure similarity between vectors — high dot product means the vectors point in a similar direction (similar content). Attention mechanisms in transformers and recommendation systems both rely heavily on dot products to compare meaning and relevance.

A derivative measures how much a function's output changes when you change its input slightly — it's the slope at a specific point. In ML, derivatives tell us how to adjust model parameters to reduce the prediction error. If increasing a weight by a tiny amount reduces the error, the derivative is negative (go that way). If it increases the error, the derivative is positive (go the other way). Training is essentially following derivatives to find the best settings.

Gradient descent is the algorithm that trains most ML models. Imagine you're blindfolded on a hilly landscape and trying to reach the lowest valley — you feel the slope around you and step in the downhill direction. That's gradient descent: the 'hill' is the error, and the algorithm adjusts model parameters step by step to minimize it. Each step size is controlled by a setting called the 'learning rate.'

The learning rate controls how big each adjustment step is during training. Too large and the model overshoots the minimum — bouncing around and never converging. Too small and training takes forever, potentially getting stuck. A well-chosen learning rate finds the bottom of the error curve efficiently. Techniques like learning rate scheduling gradually reduce the rate over training — starting bold and getting more precise as you approach the optimum.

A loss function measures how wrong the model's predictions are — it's the score that training tries to minimize. For regression, a common loss is mean squared error (average squared difference between predicted and real values). For classification, cross-entropy loss measures how far off the probability predictions are. The loss function is what gradient descent is trying to make as small as possible during training.

Mean squared error calculates how wrong a model's predictions are on average. For each prediction, subtract the actual value, square the result (to make negatives positive and penalize big errors more), then average across all examples. If a model predicts house prices as $300k, $250k, $400k but the real prices are $320k, $240k, $380k, MSE penalizes each error. Squaring makes large mistakes far more expensive than small ones.

Linear regression finds the best-fitting straight line through data points that connects an input (feature) to an output (prediction). If you plot students' study hours on the x-axis and grades on the y-axis, linear regression draws the line that minimizes the distance between the line and all data points. You can then use the line to predict: 'If a student studies 6 hours, what grade would they likely get?'

The sigmoid function takes any number and squeezes it to a value between 0 and 1 — making it perfect for outputting probabilities. Its formula is 1/(1+e^(-x)). Very large inputs give outputs near 1; very negative inputs give outputs near 0. In binary classification (spam or not?), the final neural network layer uses sigmoid to convert a raw score into a probability. Logistic regression uses sigmoid as its core function.

Logarithms compress very large or very small numbers into a manageable range, making math more stable. In ML, the cross-entropy loss function uses the natural logarithm to penalize confident wrong predictions very heavily. Probabilities like 0.0001 or 0.99999 appear in neural networks, and log transforms them into values like -9.2 and -0.00001 — much easier to work with mathematically. 'Log loss' is a standard metric for classification models.

Euclidean distance is the straight-line distance between two points — the 'as the crow flies' distance. In a 2D space, for points (1,2) and (4,6), distance = √((4-1)²+(6-2)²) = √(9+16) = √25 = 5. ML algorithms like k-nearest neighbors (KNN) use Euclidean distance to find which training examples are 'closest' to a new input, then predict based on those neighbors' labels.

Cosine similarity measures the angle between two vectors — not their length, just their direction. Two word vectors pointing in nearly the same direction have high cosine similarity (close to 1), meaning the words have similar meanings. 'King' and 'Queen' are more similar in direction than 'King' and 'Bicycle.' Search engines and recommendation systems use cosine similarity to find semantically similar documents or items.

The independent variable (also called a feature or input) is what you use to make a prediction. The dependent variable (label or target) is what you're trying to predict. In a house-price model: square footage, bedrooms, and location are independent variables; the sale price is the dependent variable. The model learns how changes in independent variables affect the dependent variable — exactly the same concept as in school science experiments.

Normalization scales features to a common range — typically 0 to 1 or mean 0 with standard deviation 1 — so no feature dominates just because of its unit of measurement. Height in centimeters (150–200) and income in dollars (20,000–200,000) are very different scales. Without normalization, gradient descent takes much longer to converge because it has to navigate a very uneven landscape. Normalizing first makes training faster and often produces better models.

A confusion matrix is a 2×2 table (for binary classification) counting: true positives (correctly predicted positive), false positives (incorrectly said positive), true negatives (correctly predicted negative), and false negatives (missed positives). These four numbers let you calculate accuracy, precision, recall, and F1 score. For a disease test: true positives are correct diagnoses; false negatives are missed cases — the most dangerous type of error in medicine.

F1 score is the harmonic mean of precision and recall — a single number that balances both. It's especially useful when your dataset is imbalanced (many more of one class than another). If you have 99% legitimate transactions and 1% fraud, a model that classifies everything as legitimate gets 99% accuracy but 0% F1 score for fraud detection. F1 forces the model to actually catch real cases rather than just predicting the majority class.

Bayes' theorem is a formula for updating your belief about something when you get new evidence. If you know 1% of emails are spam, and you see the word 'FREE MONEY!!!' in an email, Bayes lets you calculate the updated probability it's spam given that specific evidence. It says: new belief = (how well evidence supports the hypothesis) × (prior belief) ÷ (probability of seeing this evidence overall). It's the foundation of many ML classifiers.

Regularization is a technique that prevents overfitting by penalizing overly complex models during training. Without it, a model might memorize the training data perfectly but fail on new data. Regularization adds a penalty to the loss function for large parameter values. L1 regularization can drive some weights all the way to zero (feature selection). L2 regularization keeps all weights small. It's like telling the model: 'Keep things simple — don't try to memorize every quirk.'

A regular parameter (like a neural network weight) is learned automatically during training. A hyperparameter is a setting you choose before training begins — like the learning rate, number of layers, or regularization strength. Hyperparameters aren't learned by the training algorithm; you set them manually or through a search process called hyperparameter tuning. Choosing good hyperparameters is one of the most important (and tedious) parts of building ML models.

Bias is error from an oversimplified model that misses real patterns (underfitting). Variance is error from an overfit model that memorizes training noise (overfitting). These two errors trade off: simpler models have high bias and low variance; complex models have low bias and high variance. The sweet spot is a model with just enough complexity to capture real patterns without memorizing noise — this tension is the fundamental challenge of ML model selection.

Dimensionality refers to the number of features (columns) in your dataset. A dataset with 3 features is 3-dimensional; one with 1,000 features is 1,000-dimensional. High dimensionality is common (genetics datasets can have millions of features) and causes a problem called the 'curse of dimensionality' — data becomes sparse, distances lose meaning, and models need exponentially more examples to train well. Dimensionality reduction (like PCA) compresses features to manageable numbers.

PCA is a dimensionality reduction technique that finds the directions in data where variation is greatest and projects the data onto a smaller number of those directions. Imagine spreading gravel on a floor — PCA finds the main 'spread' directions and represents each grain's position using just those two directions instead of all three (x, y, z). It compresses high-dimensional data while preserving the most important patterns, enabling faster training and visualization.

The correlation coefficient (r) measures how strongly two variables are linearly related. r = 1 means a perfect positive relationship (as one goes up, so does the other exactly). r = -1 means perfect negative relationship. r = 0 means no linear relationship at all. In ML, correlation helps with feature selection — highly correlated features often carry redundant information and one can be removed to simplify the model without losing much predictive power.

One-hot encoding converts categorical variables into numerical format by creating a new binary column for each category. 'Fruit: Apple/Banana/Cherry' becomes three columns: Apple=[1,0,0], Banana=[0,1,0], Cherry=[0,0,1]. ML algorithms need numbers, not words. One-hot encoding is the standard way to give categories equal mathematical 'distance' from each other — so the model doesn't think Cherry is 'three times' Apple just because 3 > 1 in a simple numeric encoding.

Softmax converts a list of numbers into probabilities that sum to 1 — perfect for multiclass classification where you need to output a probability for each possible category. If a model classifies an image as cat, dog, or bird, softmax turns the raw scores [2.0, 1.0, 0.1] into probabilities [0.70, 0.26, 0.04] — 70% cat, 26% dog, 4% bird. The highest probability wins. Softmax is almost always the last layer in a multiclass neural network.

Matrix multiplication combines two matrices to produce a third. For an ML layer, the operation is: output = weights × input. Every single layer in a neural network is essentially a matrix multiplication followed by an activation function. Modern GPUs are designed specifically to do matrix multiplications extremely fast in parallel — a single GPU can do trillions of such operations per second. Deep learning's power comes from stacking many of these efficient matrix operations.

Exponential growth means something doubles repeatedly over equal intervals — 1, 2, 4, 8, 16, 32... In AI, two things grow exponentially: the size of language models (parameters went from millions to trillions in a few years) and the computing power used to train them. Understanding exponential growth helps you appreciate why AI has progressed so rapidly recently — it's not linear improvement, it's compounding. But exponential compute costs also explain why only a few organizations can train frontier models.

Entropy from information theory measures uncertainty or randomness in a set of outcomes. High entropy means the distribution is very uncertain (many equally likely outcomes). Low entropy means one outcome is dominant. In decision trees, each split aims to reduce entropy — dividing data so each group becomes more 'pure' (dominated by one class). The split that reduces entropy the most is chosen. Cross-entropy loss in neural networks is also derived from this concept.

The Pythagorean theorem is used whenever we calculate Euclidean distance — the straight-line distance between two points. In high-dimensional feature spaces, distance = √(Σ(a_i - b_i)²), which is a generalization of a²+b²=c². Algorithms like k-nearest neighbors (KNN) and k-means clustering rely on this distance formula. It also appears in the calculation of vector magnitudes used in cosine similarity. Ancient Greek geometry turns out to be fundamental to modern AI.

A gradient is a vector of all partial derivatives of a function with respect to each of its input variables. In ML, it points in the direction of the steepest uphill climb in the error surface. Gradient descent subtracts a fraction of the gradient from each weight — moving downhill. With millions of parameters, computing gradients efficiently requires the backpropagation algorithm, which applies the chain rule of calculus to propagate error signals backward through the network.

The chain rule from calculus lets you find the derivative of a 'nested' function — a function inside another function. A deep neural network is a huge nested function: input passes through layer 1, layer 2, layer 3... The chain rule lets us efficiently calculate how much each weight (even in early layers) contributed to the final error. Without it, training deep networks would be computationally impossible. Backpropagation is just the chain rule applied efficiently.

Absolute value |x| is the distance from zero — always non-negative. |-5| = 5, |3| = 3. In ML, absolute values appear in L1 loss (mean absolute error), which calculates the average |prediction - actual| across all examples. Compared to mean squared error (MSE), L1 loss is less sensitive to extreme outliers because it doesn't square the errors. It's often preferred for regression tasks where outliers shouldn't have outsized influence.

These are mathematical objects of increasing dimensionality. A scalar is a single number (temperature: 37°C). A vector is a list of numbers (one data point with features [3, 1500, 2]). A matrix is a 2D grid of numbers (a dataset). A tensor is a generalization to any number of dimensions — a color image is a 3D tensor (height × width × 3 color channels); a batch of images is a 4D tensor. Neural networks process tensors at every step.

Modular arithmetic (clock math) finds the remainder after division. 17 mod 5 = 2 (since 17 = 3×5 + 2). In AI research, modular arithmetic appears in a famous example of 'grokking': small transformers trained to do modular addition first memorize training examples, then suddenly 'generalize' to all cases — revealing interesting insights about how neural networks learn abstract mathematical structure, not just patterns in data.

Expected value is the average outcome you'd expect over many repetitions of a random event. For a fair die: E = (1+2+3+4+5+6)/6 = 3.5. In ML, expected value appears everywhere — the loss function is the expected loss over the training data distribution. Reinforcement learning agents maximize expected cumulative reward. Decision trees split to maximize expected information gain. It's the mathematical foundation for reasoning about uncertain outcomes.

The transpose of a matrix flips it along its diagonal — rows become columns and columns become rows. A 3×4 matrix becomes 4×3 after transposing. Transpose is used constantly in neural network math: the backward pass in backpropagation uses transposed weight matrices to propagate gradients. In Python with NumPy, you transpose with '.T': matrix.T. It's one of the most-used linear algebra operations in all of deep learning.

Sigmoid squashes outputs to (0,1) — nice for probabilities but causes vanishing gradients in deep networks since its gradient is nearly zero at extreme values. ReLU (max(0,x)) has gradient either 0 or 1, so gradients don't vanish for positive activations — making training much more stable in deep networks. This is why ReLU replaced sigmoid in almost all hidden layers of modern deep networks. Sigmoid is still used in the final output layer for binary classification.

The normal (Gaussian) distribution describes data that clusters symmetrically around a mean, with fewer examples further from the center. Many natural phenomena (heights, test scores, measurement errors) roughly follow it. In ML, neural network weights are often initialized from a normal distribution. Gaussian noise is added during data augmentation. The normal distribution's mathematical properties make many statistical techniques tractable. Recognizing when data is approximately normal helps choose appropriate models.

Log loss for a binary classifier is: -(y × log(p) + (1-y) × log(1-p)), where y is the true label (0 or 1) and p is the predicted probability. When the model correctly predicts high confidence (y=1, p=0.95), log(0.95) is near 0 — small loss. When it's confidently wrong (y=1, p=0.05), log(0.05) ≈ -3 — a huge loss. Squaring the error (MSE) would be gentler; log loss brutally punishes confident mistakes, making models well-calibrated.

The determinant is a single number that encodes geometric properties of a square matrix — specifically whether it can be inverted and by how much it scales volume. In classic statistics, determinants appear in multivariate Gaussian distributions and in computing matrix inverses needed for certain regression formulas. In modern deep learning, direct matrix inverses are rarely used (too expensive for huge matrices), but determinants appear in flow-based generative models and Gaussian process research.

The derivative of the sigmoid function is σ(x)×(1-σ(x)). This is maximized at x=0 (value = 0.25) and approaches 0 for large positive or negative inputs. In a deep network using sigmoid in every layer, multiplying these tiny gradients together in backpropagation causes them to vanish exponentially — each layer gets a weaker and weaker update signal. This 'vanishing gradient' is why ReLU (whose derivative is simply 0 or 1) made training deep networks practical.

Polynomial features transform input variables by adding higher-power versions: x → x, x², x³. A model that is linear in the original feature becomes nonlinear when polynomial features are added, letting it fit curves instead of just straight lines. If test scores have a curve relationship with study hours (initially improving fast, then leveling off), adding x² captures that curve. Too many polynomial features cause overfitting — the model memorizes curves in the training data that don't generalize.