Linear algebra for ML

A scalar is a single number (0-dimensional), a vector is an ordered list of numbers (1-dimensional, like a row or column), and a matrix is a rectangular grid of numbers arranged in rows and columns (2-dimensional). In ML, a scalar might be a single weight, a vector might represent one data sample's features, and a matrix might hold a whole dataset.

The shape is (100, 5), meaning 100 rows (one per sample) and 5 columns (one per feature). By convention in ML, rows usually represent samples and columns represent features. So X has 100 x 5 = 500 individual numbers.

import numpy as np
X = np.zeros((100, 5))
print(X.shape)  # (100, 5)

The dot product multiplies corresponding entries of two vectors and sums the results, giving a single scalar. Geometrically it equals |a||b|cos(theta), where theta is the angle between them, so it measures how much the two vectors point in the same direction. A dot product of zero means the vectors are perpendicular (orthogonal).

import numpy as np
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
print(np.dot(a, b))  # 32

To multiply matrix A by matrix B (computing AB), the number of columns of A must equal the number of rows of B. If A is (m x n) and B is (n x p), the product AB has shape (m x p). The inner dimensions must match and they 'cancel out' to leave the outer dimensions.

import numpy as np
A = np.random.rand(2, 3)
B = np.random.rand(3, 4)
print((A @ B).shape)  # (2, 4)

No, matrix multiplication is generally NOT commutative, meaning AB is usually not equal to BA. In fact, BA may not even be defined if the shapes don't conform, and even for square matrices the two products usually differ. This is unlike scalar multiplication where order does not matter.

The identity matrix I is a square matrix with 1s on the main diagonal and 0s elsewhere. It acts like the number 1 for matrices: for any compatible matrix A, AI = IA = A. It is essential in defining matrix inverses, since A times A-inverse equals I.

import numpy as np
print(np.eye(3))

The transpose of a matrix A, written A^T, flips it over its main diagonal, turning rows into columns and columns into rows. If A has shape (m x n), then A^T has shape (n x m). The element at position (i, j) in A becomes the element at (j, i) in A^T.

import numpy as np
A = np.array([[1, 2, 3], [4, 5, 6]])
print(A.T.shape)  # (3, 2)

X is the data matrix (samples x features), w is the weight vector (one weight per feature), and b is a bias term. The product Xw computes, for each sample, the weighted sum of its features, producing a prediction vector to which b is added. This expresses many predictions at once compactly as a matrix-vector multiplication.

import numpy as np
X = np.random.rand(4, 3)
w = np.random.rand(3)
b = 0.5
y = X @ w + b
print(y.shape)  # (4,)

The L2 norm of a vector is the square root of the sum of the squares of its components, representing its length or magnitude. In ML it measures the size of vectors such as weight vectors or error vectors, and it underlies Euclidean distance between points. L2 regularization penalizes large weights using the squared L2 norm.

import numpy as np
v = np.array([3, 4])
print(np.linalg.norm(v))  # 5.0

The L1 norm is the sum of the absolute values of a vector's components, while the L2 norm is the square root of the sum of squares. L1 regularization tends to push some weights exactly to zero, producing sparse models, whereas L2 shrinks weights smoothly toward zero without forcing exact zeros. This makes L1 useful for feature selection.

import numpy as np
w = np.array([1, -2, 3])
print(np.sum(np.abs(w)))  # L1 = 6

Two vectors are orthogonal if they are perpendicular, meaning the angle between them is 90 degrees. You check this by computing their dot product: if the dot product is zero, the vectors are orthogonal. Orthogonality is important in ML for ideas like uncorrelated features and orthogonal bases (e.g., in PCA).

import numpy as np
a = np.array([1, 0])
b = np.array([0, 1])
print(np.dot(a, b) == 0)  # True

A linear combination of vectors is formed by multiplying each vector by a scalar coefficient and adding the results together. For example, 2v1 + 3v2 - v3 is a linear combination of v1, v2, and v3. This concept underlies span, basis, and how a linear model combines feature vectors.

The span of a set of vectors is the collection of all possible linear combinations of those vectors. Geometrically, the span of two non-parallel vectors in 3D is a plane, and the span of a single nonzero vector is a line. It describes all the points you can 'reach' using those vectors.

A set of vectors is linearly independent if no vector in the set can be written as a linear combination of the others, meaning none is redundant. Equivalently, the only way to combine them to get the zero vector is with all coefficients equal to zero. Linear independence ensures features carry non-redundant information.

The rank of a matrix is the number of linearly independent rows (equivalently columns) it has. It tells you the true dimensionality of the information in the matrix. In ML, a low-rank data matrix indicates redundant or correlated features, which is relevant for dimensionality reduction and for whether systems of equations have unique solutions.

import numpy as np
A = np.array([[1, 2], [2, 4]])
print(np.linalg.matrix_rank(A))  # 1

The inverse of a square matrix A, written A^-1, is the matrix such that A times A^-1 equals the identity matrix I. An inverse exists only when the matrix is square and non-singular, meaning its determinant is nonzero (equivalently, it has full rank). Singular matrices have no inverse.

import numpy as np
A = np.array([[2, 0], [0, 4]])
print(np.linalg.inv(A))

The determinant of a square matrix measures how the linear transformation it represents scales area (in 2D) or volume (in 3D). A determinant of zero means the transformation collapses space into a lower dimension, so the matrix is singular and non-invertible. A negative determinant indicates the orientation is flipped.

import numpy as np
A = np.array([[1, 2], [3, 4]])
print(np.linalg.det(A))  # -2.0

An eigenvector of a square matrix A is a nonzero vector v whose direction is unchanged when A is applied to it; it only gets scaled. The scaling factor is the eigenvalue lambda, satisfying Av = lambda*v. Eigenvectors reveal the 'special directions' of a transformation and are central to PCA.

import numpy as np
A = np.array([[2, 0], [0, 3]])
vals, vecs = np.linalg.eig(A)
print(vals)  # [2. 3.]

PCA finds new axes (principal components) that capture the directions of maximum variance in the data. These components are the eigenvectors of the data's covariance matrix, and the corresponding eigenvalues indicate how much variance each direction explains. By keeping only the top components, PCA reduces dimensionality while preserving most of the information.

Element-wise (Hadamard) multiplication multiplies matrices entry by entry and requires both matrices to have the same shape, producing a result of that same shape. Matrix multiplication instead involves dot products of rows and columns and requires the inner dimensions to match. In NumPy, * does element-wise and @ does matrix multiplication.

import numpy as np
A = np.array([[1, 2], [3, 4]])
B = np.array([[10, 20], [30, 40]])
print(A * B)   # element-wise
print(A @ B)   # matrix mult

Broadcasting is NumPy's mechanism for automatically expanding arrays of different but compatible shapes so they can be combined element-wise without explicitly copying data. For example, adding a (1 x n) bias vector to an (m x n) matrix applies the bias to every row. This lets you write concise, efficient code for operations like adding a bias to all samples.

import numpy as np
X = np.ones((3, 2))
b = np.array([10, 20])
print(X + b)  # b added to each row

The dot product is large and positive when two vectors point in similar directions, so it serves as a basic similarity measure. Cosine similarity refines this by dividing the dot product by the product of the vectors' lengths, giving a value between -1 and 1 that depends only on direction, not magnitude. This makes it useful for comparing documents or embeddings regardless of their size.

import numpy as np
a = np.array([1, 1])
b = np.array([2, 2])
cos = np.dot(a, b)/(np.linalg.norm(a)*np.linalg.norm(b))
print(cos)  # 1.0

A diagonal matrix has nonzero entries only on its main diagonal, with zeros everywhere else. Multiplying by a diagonal matrix simply scales each coordinate independently, and operations like finding its inverse or determinant are trivial (just use the diagonal entries). This simplicity makes diagonal matrices efficient and common in scaling and normalization steps.

import numpy as np
print(np.diag([2, 5, 7]))

A symmetric matrix equals its own transpose, so the entry at (i, j) equals the entry at (j, i). Covariance matrices and Gram matrices are symmetric, which is important because symmetric matrices have real eigenvalues and orthogonal eigenvectors. This property is exploited in PCA and many optimization methods.

import numpy as np
A = np.array([[1, 2], [2, 3]])
print(np.array_equal(A, A.T))  # True

The mean squared error loss is computed over all samples at once, and its gradient with respect to the weights is the vector of partial derivatives. Using matrices, this gradient can be written compactly, for example as proportional to X^T (Xw - y), which computes all partial derivatives in a single operation. This vectorized form is both mathematically clean and computationally fast.

import numpy as np
# grad of MSE wrt w (up to constant)
# grad = X.T @ (X @ w - y)

The normal equation gives the optimal weights directly as w = (X^T X)^-1 X^T y, without iterative optimization. It requires multiplying matrices, transposing X, and inverting the square matrix X^T X. This inverse exists only when X^T X is non-singular (the features are linearly independent).

import numpy as np
# w = np.linalg.inv(X.T @ X) @ X.T @ y

Vectorized operations use optimized, low-level libraries (like BLAS underneath NumPy) that process whole arrays at once, making them far faster than explicit Python loops. They also make code shorter and closer to the mathematical notation. For large datasets, this speed difference can be orders of magnitude.

import numpy as np
a = np.arange(1_000_000)
# vectorized sum is far faster than a Python for-loop
print(a.sum())

The trace of a square matrix is the sum of the entries on its main diagonal. It equals the sum of the matrix's eigenvalues and is invariant under similarity transformations. The trace appears in various ML derivations and in measuring quantities like total variance.

import numpy as np
A = np.array([[1, 2], [3, 4]])
print(np.trace(A))  # 5

One-hot encoding represents a category as a vector that has a 1 in the position corresponding to that category and 0s everywhere else. For example, with three classes, class 2 becomes [0, 1, 0]. This turns non-numeric labels into vectors that linear models and neural networks can process through matrix multiplication.

import numpy as np
# class index 2 of 4 classes
onehot = np.eye(4)[2]
print(onehot)  # [0. 0. 1. 0.]

A dense layer computes its output as an affine transformation: it multiplies the input vector (or batch matrix) by a weight matrix and adds a bias vector, then applies a nonlinear activation. The matrix multiplication mixes the inputs in learned ways to produce each output neuron's pre-activation value. So the core of every dense layer is matrix-vector (or matrix-matrix) multiplication.

import numpy as np
x = np.array([1.0, 2.0])
W = np.array([[0.5, -0.5], [1.0, 0.0]])
b = np.array([0.1, 0.2])
print(W @ x + b)

A row vector has shape (1 x n) and a column vector has shape (n x 1); they hold the same numbers but are oriented differently. The orientation matters because matrix multiplication depends on shapes: a row vector times a column vector gives a scalar (dot product), while a column times a row gives an outer-product matrix. Mismatched orientations can cause shape errors or unintended results.

import numpy as np
row = np.array([[1, 2, 3]])      # (1,3)
col = np.array([[1], [2], [3]]) # (3,1)
print((row @ col).shape)  # (1,1)

The outer product of an m-dimensional vector and an n-dimensional vector is an (m x n) matrix where entry (i, j) is the product of the i-th element of the first vector and the j-th element of the second. Unlike the dot product, which returns a scalar, the outer product returns a full matrix. It appears in rank-one updates and in computing certain gradients.

import numpy as np
a = np.array([1, 2])
b = np.array([3, 4, 5])
print(np.outer(a, b).shape)  # (2, 3)

The covariance matrix is a symmetric matrix where the diagonal entries are the variances of individual features and the off-diagonal entries are the covariances between pairs of features. A positive off-diagonal value means two features tend to increase together, while a negative one means one rises as the other falls. It summarizes how features vary and co-vary, and is the basis for PCA.

import numpy as np
X = np.random.rand(100, 3)
print(np.cov(X, rowvar=False).shape)  # (3, 3)

When features are on very different scales, distance-based methods are dominated by the largest-magnitude feature, and gradient descent takes inefficient zig-zag steps. Scaling features to comparable ranges (e.g., zero mean and unit variance) makes distances fairer and the loss surface more balanced, speeding up convergence. Linear algebra views this as transforming the data so its covariance is better conditioned.

import numpy as np
X = np.array([[1.0, 1000.0], [2.0, 2000.0]])
X_scaled = (X - X.mean(0)) / X.std(0)
print(X_scaled)

A singular matrix is a square matrix that has no inverse, which happens exactly when its determinant is zero. This occurs when its rows (or columns) are linearly dependent, so the matrix does not have full rank. In ML, perfectly correlated features can make X^T X singular, breaking the normal-equation solution.

import numpy as np
A = np.array([[1, 2], [2, 4]])
print(np.linalg.det(A))  # 0.0 -> singular

SVD factors any matrix A into three matrices U, Sigma, and V^T, where U and V contain orthogonal directions and Sigma is a diagonal matrix of non-negative singular values that measure importance. By keeping only the largest singular values, you get the best low-rank approximation of the matrix. This powers dimensionality reduction, data compression, and recommendation systems.

import numpy as np
A = np.random.rand(4, 3)
U, S, Vt = np.linalg.svd(A)
print(S.shape)  # (3,)

The Euclidean distance between two points is the L2 norm of the difference of their position vectors: you subtract the vectors, square each component of the result, sum them, and take the square root. This generalizes the Pythagorean theorem to any number of dimensions. It is widely used in algorithms like k-nearest neighbors and k-means.

import numpy as np
p = np.array([0, 0])
q = np.array([3, 4])
print(np.linalg.norm(p - q))  # 5.0

Projecting vector a onto vector b gives the component of a that lies in the direction of b, computed as (a dot b / b dot b) times b. It answers 'how much of a points along b'. Projections underlie least-squares regression (projecting targets onto the column space of features) and dimensionality reduction like PCA.

import numpy as np
a = np.array([2, 3])
b = np.array([1, 0])
proj = (np.dot(a, b)/np.dot(b, b)) * b
print(proj)  # [2. 0.]

In a (samples x features) matrix, summing or averaging along axis 0 collapses the rows to give a per-feature statistic, while axis 1 collapses the columns to give a per-sample statistic. Choosing the wrong axis computes the wrong quantity (e.g., averaging across features instead of across samples). Understanding axes is essential for correct preprocessing like computing feature means.

import numpy as np
X = np.array([[1, 2], [3, 4]])
print(X.sum(axis=0))  # [4 6] per feature
print(X.sum(axis=1))  # [3 7] per sample

Multiplying a vector by a matrix applies a linear transformation that can rotate, scale, reflect, or shear the vector while keeping the origin fixed and lines straight. The columns of the matrix tell you where the standard basis vectors land after the transformation. This view explains why neural network layers and many ML operations are geometric transformations of data.

import numpy as np
R = np.array([[0, -1], [1, 0]])  # 90-degree rotation
v = np.array([1, 0])
print(R @ v)  # [0 1]