Unsupervised learning algorithms

Unsupervised learning works with data that has no labels or target outputs; the algorithm must discover hidden structure (such as groups or patterns) on its own. Supervised learning, by contrast, learns a mapping from inputs to known labels. Because there are no 'correct answers' to compare against, evaluating unsupervised models is often harder and more subjective.

Clustering aims to group data points so that points within the same group (cluster) are more similar to each other than to points in other groups. It is an unsupervised task because no predefined labels guide the grouping. Common uses include customer segmentation and grouping similar documents.

K is the number of clusters the algorithm will partition the data into, and it must be chosen by the user before training. K-means then finds K centroids and assigns each point to its nearest centroid. Choosing a poor K can lead to under- or over-segmenting the data.

from sklearn.cluster import KMeans
km = KMeans(n_clusters=3, random_state=0)
labels = km.fit_predict(X)

K-means alternates between an assignment step and an update step. In the assignment step, each data point is assigned to the nearest centroid; in the update step, each centroid is recomputed as the mean of the points assigned to it. These steps repeat until assignments no longer change (convergence).

K-means minimizes the within-cluster sum of squared distances between each point and its assigned centroid, often called inertia or WCSS. Lower inertia means points are tighter around their centroids. However, inertia always decreases as K increases, so it cannot alone be used to pick K.

The elbow method helps choose a good number of clusters K for K-means. You plot inertia (within-cluster sum of squares) against K and look for the 'elbow' point where adding more clusters yields only small improvements. That bend suggests a reasonable trade-off between simplicity and fit.

inertias = []
for k in range(1, 10):
    km = KMeans(n_clusters=k).fit(X)
    inertias.append(km.inertia_)

K-means relies on distance calculations, so features with larger numeric ranges dominate the distance and distort the clusters. Scaling (e.g., standardization) puts all features on a comparable scale so each contributes fairly. Without scaling, a feature like 'income in rupees' could overwhelm 'age in years'.

from sklearn.preprocessing import StandardScaler
Xs = StandardScaler().fit_transform(X)

Plain K-means picks initial centroids randomly, which can lead to poor or unstable clusterings. K-means++ chooses initial centroids that are spread out, making the algorithm converge faster and more reliably to better solutions. It is the default initialization in many libraries like scikit-learn.

K-means is sensitive to its initial centroid placement and can converge to different local minima depending on where it starts. Random initialization therefore can produce different final clusters across runs. Setting a random seed or using K-means++ and multiple restarts reduces this variability.

A centroid is the center point of a cluster, computed as the mean (average position) of all data points assigned to that cluster. K-means uses centroids to represent clusters and assigns each point to the nearest one. Centroids are updated each iteration as cluster membership changes.

Hierarchical clustering builds a tree of nested clusters, either by merging points bottom-up (agglomerative) or splitting top-down (divisive). Its results are commonly visualized with a dendrogram, a tree diagram showing how and at what distance clusters merge. You can 'cut' the dendrogram at a chosen height to obtain a specific number of clusters.

A linkage method defines how the distance between two clusters is measured when deciding which clusters to merge. Examples include single linkage (closest pair of points), complete linkage (farthest pair), and average linkage (average of all pairwise distances). The choice of linkage strongly affects the shape of the resulting clusters.

DBSCAN can find clusters of arbitrary (non-spherical) shapes and automatically identifies outliers as noise, without requiring you to specify the number of clusters in advance. K-means, by contrast, assumes roughly spherical clusters and needs K to be set beforehand. This makes DBSCAN useful for spatial data with irregular cluster shapes.

from sklearn.cluster import DBSCAN
labels = DBSCAN(eps=0.5, min_samples=5).fit_predict(X)

eps is the maximum distance for two points to be considered neighbors, and min_samples is the minimum number of points needed within that distance to form a dense region (core point). Together they define what counts as a dense cluster versus noise. Tuning them well is essential because they directly determine how many clusters and outliers DBSCAN produces.

DBSCAN explicitly labels points that lie in low-density regions as noise (outliers) and does not force them into any cluster. K-means assigns every point to some cluster, so outliers get pulled into the nearest cluster and can distort its centroid. This makes DBSCAN more robust to noisy data.

PCA is a dimensionality-reduction technique that transforms data into a smaller set of new features called principal components, which capture the directions of greatest variance. It reduces the number of dimensions while preserving as much information (variance) as possible. This helps with visualization, speed, and reducing noise.

from sklearn.decomposition import PCA
X2 = PCA(n_components=2).fit_transform(X)

The first principal component is the direction in the data along which the variance is largest. It is the single axis that captures the most spread (information) in the dataset. Subsequent components each capture the next-largest variance while being orthogonal to the previous ones.

Orthogonality ensures that each principal component captures information not already captured by the others, so the components are uncorrelated. This avoids redundancy and lets each new axis explain a distinct portion of the variance. As a result, a few orthogonal components can summarize the data efficiently.

The explained variance ratio gives the fraction of the dataset's total variance captured by each principal component. Summing these ratios over the components you keep tells you how much information is retained after dimensionality reduction. It helps decide how many components are enough (e.g., enough to retain 95% of variance).

pca = PCA().fit(X)
print(pca.explained_variance_ratio_)

PCA computes the eigenvectors and eigenvalues of the data's covariance matrix. The eigenvectors give the directions of the principal components, and the eigenvalues indicate how much variance each direction explains. Components are ranked by their eigenvalues from largest to smallest.

As the number of features (dimensions) grows, data points become sparse and distances between them become less meaningful, since most points end up roughly equidistant. This degrades distance-based methods like K-means and nearest-neighbor approaches. Dimensionality reduction such as PCA is one way to mitigate it.

The silhouette score measures how well each point fits its own cluster compared to other clusters, ranging from -1 to +1. Values near +1 mean points are well matched to their cluster and far from others, near 0 means overlapping clusters, and negative means points may be in the wrong cluster. It is a common label-free way to evaluate clustering quality.

from sklearn.metrics import silhouette_score
score = silhouette_score(X, labels)

In hard clustering, each data point belongs to exactly one cluster (e.g., K-means). In soft or fuzzy clustering, each point is assigned a probability or degree of membership to multiple clusters (e.g., Gaussian Mixture Models). Soft clustering is useful when boundaries between groups are not crisp.

A GMM assumes the data is generated from a mixture of several Gaussian (normal) distributions, each with its own mean and covariance. It performs soft clustering by estimating the probability that each point came from each Gaussian component. This lets it model elliptical clusters of different sizes and orientations, unlike basic K-means.

from sklearn.mixture import GaussianMixture
gm = GaussianMixture(n_components=3).fit(X)

Association rule mining discovers interesting relationships between items in large datasets, such as which products are frequently bought together. A classic example is 'customers who buy bread also tend to buy butter.' It is unsupervised because it finds patterns without predicting a labeled target.

Support measures how frequently an itemset appears in the dataset, calculated as the fraction of transactions that contain that itemset. A high support means the combination is common across all transactions. It is used to filter out rare item combinations before generating rules.

Confidence measures how often the rule is correct, calculated as the fraction of transactions containing A that also contain B. In other words, it is the conditional probability of B given A. High confidence means B is very likely to be purchased when A is purchased.

A lift greater than 1 means items A and B appear together more often than expected if they were independent, suggesting a positive association between them. A lift of exactly 1 means no association, and less than 1 means they appear together less than expected. Lift therefore helps identify genuinely meaningful rules beyond mere frequency.

Anomaly detection identifies data points that differ significantly from the majority, such as fraudulent transactions or faulty sensor readings. It is often unsupervised because labeled examples of anomalies are rare or unavailable, so the model learns what 'normal' looks like and flags deviations. Techniques include clustering-based methods, isolation forests, and density estimation.

t-SNE (t-distributed Stochastic Neighbor Embedding) is a nonlinear dimensionality-reduction method used mainly for visualizing high-dimensional data in 2D or 3D. It tries to keep similar points close together in the low-dimensional map, revealing clusters. It is great for visualization but not typically used as a general-purpose feature reducer for downstream models.

from sklearn.manifold import TSNE
emb = TSNE(n_components=2).fit_transform(X)

PCA is a linear method that finds directions of maximum variance and is fast and deterministic, making it useful for both compression and visualization. t-SNE is a nonlinear method focused on preserving local neighborhoods for visualization, but it is slower, non-deterministic, and not meant for reducing features before modeling. In short, PCA preserves global variance while t-SNE emphasizes local structure.

An autoencoder is a neural network trained to compress input data into a lower-dimensional code and then reconstruct the original input from that code. Because the target is the input itself, no external labels are needed, making it unsupervised (self-supervised). The compressed middle layer learns a useful low-dimensional representation, similar in spirit to PCA but able to capture nonlinear structure.

Standard K-means uses Euclidean (straight-line) distance to measure how far each point is from each centroid. This is consistent with minimizing the squared distances, since the mean minimizes squared Euclidean distance. Because of this, K-means tends to form spherical clusters.

Standard K-means relies on computing means and Euclidean distances, which are not meaningful for categorical (non-numeric) values like color or city. Averaging categories produces nonsensical centroids. Alternatives such as K-modes (which uses matching dissimilarity and modes) are designed specifically for categorical data.

Cosine similarity is preferred when the direction (pattern) of the data matters more than its magnitude, such as with text documents represented as word-count vectors. Two documents about the same topic can have very different lengths but similar word proportions, which cosine captures well. Euclidean distance would unfairly separate them due to differing magnitudes.

Normalization rescales features to a fixed range, typically [0, 1], using the min and max values. Standardization transforms features to have a mean of 0 and a standard deviation of 1 (z-scores). Standardization is less affected by the data's range and is often preferred for distance-based and PCA methods, while min-max is useful when a bounded range is needed.

from sklearn.preprocessing import MinMaxScaler, StandardScaler
Xn = MinMaxScaler().fit_transform(X)
Xz = StandardScaler().fit_transform(X)

Adding more clusters means each centroid covers fewer, closer points, so the squared distances from points to their centroids shrink. In the extreme, if K equals the number of points, inertia becomes zero. This is exactly why inertia alone cannot select K; you look for the elbow instead of the minimum.

K-means implicitly assumes clusters are roughly spherical (convex) and of similar size, because it assigns points by nearest-centroid Euclidean distance. It therefore struggles with elongated, irregular, or nested cluster shapes. For such data, density-based methods like DBSCAN or Gaussian Mixture Models often work better.

Cluster labels can be added as a new feature to enrich a dataset, or clustering can compress data by representing groups with their centroids. It is also used to segment customers before building separate models per segment, or to detect and remove anomalous clusters. In this way, unsupervised clustering supports later supervised or analytical tasks.

Supervised models can be scored against known labels using metrics like accuracy, but unsupervised models have no ground-truth answers to compare against. Instead, we rely on internal measures (like silhouette score or inertia) or domain judgment, which are more subjective. This makes deciding whether a clustering or reduction is 'good' inherently more ambiguous.