Principal component analysis (PCA)

PCA is a dimensionality-reduction technique whose main goal is to transform a dataset with many correlated features into a smaller set of new uncorrelated features called principal components, while retaining as much of the original variance (information) as possible. It does this by finding the directions in the data along which variation is largest.

PCA is an unsupervised technique because it does not use any target labels or output values; it works only with the input features. It finds structure (directions of maximum variance) purely from the feature values themselves, making it useful even when no labels are available.

A principal component is a direction (a unit vector) in the feature space along which the data varies. The first principal component points in the direction of maximum variance, the second points in the direction of next-largest variance that is orthogonal (perpendicular) to the first, and so on. Each component is a linear combination of the original features.

Principal components are orthogonal so that each one captures a distinct, non-overlapping part of the data's variance, meaning the new components are uncorrelated with each other. This avoids redundancy: each component adds new information rather than repeating what an earlier component already explained.

PCA is based on variance, so features measured on larger scales would dominate the components simply because they have larger numerical ranges, not because they are more important. Standardizing each feature to zero mean and unit variance ensures every feature contributes fairly, so PCA captures true structure rather than artifacts of measurement units.

from sklearn.preprocessing import StandardScaler
X_scaled = StandardScaler().fit_transform(X)

Centering moves the data so its mean is at the origin, which is necessary because PCA measures variance as spread around the mean. Without centering, the directions of maximum spread would be biased by how far the data sits from the origin rather than reflecting its true internal variation.

The covariance matrix is a square matrix that summarizes how each pair of features varies together; its diagonal holds each feature's variance and its off-diagonal entries hold the covariances between features. In PCA, the principal components are found as the eigenvectors of this covariance matrix, and the amount of variance each captures is given by the corresponding eigenvalues.

Each eigenvector of the covariance matrix is a principal component (a direction), and its corresponding eigenvalue tells you how much variance the data has along that direction. Ordering the eigenvectors by decreasing eigenvalue gives the principal components from most to least important.

The explained variance ratio of a component is the fraction of the dataset's total variance that the component captures, equal to its eigenvalue divided by the sum of all eigenvalues. For example, a ratio of 0.6 means that component alone accounts for 60% of the total variation in the data.

pca.explained_variance_ratio_

A scree plot graphs the explained variance (or eigenvalue) of each principal component in decreasing order. It is used to choose how many components to keep by looking for the 'elbow', the point where the curve flattens out, since components after the elbow add little additional variance.

It means the data's variation lies mostly within a 2-dimensional subspace, so you can reduce the 10 features down to just 2 principal components while keeping 95% of the information. This makes the data far easier to visualize and process with only a small loss of variance.

The maximum number of principal components equals the smaller of the number of features and the number of samples minus one, that is min(n_features, n_samples - 1). For most datasets with many more samples than features, this means you can have at most as many components as there are original features.

Standard PCA finds only linear combinations of the original features, so each principal component is a weighted sum of the features. This means PCA can miss curved or non-linear structure in the data; for such cases a variant called Kernel PCA is used.

By reducing the number of features to a few principal components, PCA lowers the model's complexity and removes redundant or noisy dimensions. With fewer, more informative inputs, the model is less likely to memorize noise in the training data, which can improve its ability to generalize.

Because each principal component is a mixture (linear combination) of all the original features, the new axes usually do not correspond to any single real-world quantity. This makes it hard to explain what a given component 'means' in terms of the original, understandable features.

It returns a new array with the same number of rows (samples) as X but only 2 columns, where each column is the projection of the data onto one of the top two principal components. The original high-dimensional data is thus represented in 2D coordinates suitable for plotting.

from sklearn.decomposition import PCA
X_2d = PCA(n_components=2).fit_transform(X_scaled)
print(X_2d.shape)  # (n_samples, 2)

The loadings are the coefficients that show how much each original feature contributes to a principal component. A feature with a large absolute loading strongly influences that component, helping you interpret which original variables drive the direction of variation. In scikit-learn these weights are stored in pca.components_.

pca.components_  # rows are PCs, columns are feature weights

PC1 is the unit-length direction vector along which the projected data has the maximum possible variance. Every later component must be orthogonal to PC1 and captures the maximum remaining variance, so PC1 is the single most informative direction in the data.

Projecting means computing each data point's coordinates along the new principal-component directions, usually by taking the dot product of the centered data with each component vector. The result expresses every sample in terms of the principal components instead of the original features, often in fewer dimensions.

You can pass a float between 0 and 1 to n_components, and scikit-learn will automatically keep the smallest number of components whose cumulative explained variance reaches that fraction. For example, PCA(n_components=0.95) keeps just enough components to retain 95% of the variance.

pca = PCA(n_components=0.95)
X_reduced = pca.fit_transform(X_scaled)

Yes. Noise often appears in the low-variance directions, so by keeping only the top principal components (high variance) and discarding the rest, then reconstructing the data, much of the random noise is filtered out. This is called PCA-based denoising or reconstruction.

Humans can only easily view data in 2 or 3 dimensions, but real datasets often have many features. PCA compresses high-dimensional data into its 2 or 3 most informative components, letting you plot complex data and visually spot clusters or patterns while keeping most of the variance.

The feature with the larger numeric variance will dominate the first principal component, even if it is not actually more informative. The PCA result then reflects measurement scale rather than true structure, which is why standardizing features to comparable scales is recommended in such cases.

The total variance of the dataset equals the sum of all the eigenvalues of the covariance matrix. Therefore the proportion of variance explained by any component is its eigenvalue divided by this total sum.

Feature selection keeps a subset of the original features and discards the rest, so the chosen features remain interpretable. PCA instead creates entirely new features (principal components) by combining all the originals, which can capture variance more efficiently but loses the direct meaning of the original variables.

You reconstruct by using inverse_transform, which maps the reduced components back to the original feature space. It is only approximate because the components you dropped carried some variance, so that discarded information cannot be recovered, leaving a small reconstruction error.

X_approx = pca.inverse_transform(pca.transform(X_scaled))

PCA will combine them into essentially a single principal component, because perfectly correlated features carry the same information and lie along one direction. The redundant second direction will have a near-zero eigenvalue, so PCA effectively removes the duplication.

Yes, PCA is sensitive to outliers because it is based on variance, and a few extreme points can greatly inflate the variance in their direction. This can pull the principal components toward the outliers, distorting the representation of the bulk of the data.

Cumulative explained variance is the running total of the explained variance ratios as you add components one by one. You choose the number of components at the point where this cumulative value reaches an acceptable threshold (such as 0.95), ensuring you retain most of the information with the fewest components.

import numpy as np
np.cumsum(pca.explained_variance_ratio_)

Decorrelated (uncorrelated) features remove redundancy, so each component contributes independent information. This can improve the stability and performance of some models, such as linear regression, which can behave poorly when input features are highly correlated (multicollinearity).

PCA should be fitted only on the training set, then the same fitted transformation is applied to the test set. Fitting on test data too would leak information from the test set into the model, giving an overly optimistic and invalid estimate of performance.

pca.fit(X_train)
X_train_p = pca.transform(X_train)
X_test_p = pca.transform(X_test)

Standard PCA fails to capture non-linear structure, such as data lying on a curved manifold (for example points arranged in a spiral or concentric circles). Because PCA only finds straight-line directions of variance, non-linear methods like Kernel PCA or t-SNE are needed for such patterns.

The covariance matrix is symmetric (the covariance of A with B equals that of B with A), and a key result in linear algebra states that a symmetric matrix always has a set of mutually orthogonal eigenvectors. This is exactly why PCA's principal components come out perpendicular to one another.

It suggests the data is essentially one-dimensional: almost all the variation lies along a single direction, so the other features are largely redundant or contribute little spread. The dataset can be reduced to just one principal component with negligible loss of information.