Grouping data with clustering

Clustering is a way computers automatically group similar data points together without being told the correct groups in advance. It belongs to unsupervised learning because no one labels the data beforehand. For example, a computer can sort thousands of songs into similar-sounding groups all on its own, without any human tagging each song first.

Clustering belongs to unsupervised learning, where the algorithm finds patterns in data without any labeled examples to guide it. Unlike supervised learning, where you supply input-output pairs, clustering discovers structure entirely on its own. Think of it like sorting a jar of mixed foreign coins by shape and size when you have no idea which country they came from.

A centroid is the center point of a cluster — it is the average position of all data points assigned to that group. In K-Means, the algorithm places centroids and adjusts them repeatedly until they settle at the best central positions. You can picture a centroid as a team captain standing at the exact middle of their group of players.

The 'K' in K-Means stands for the number of clusters (groups) you want the algorithm to find. You must choose K before running the algorithm — for example, K=3 tells the computer to find exactly three groups. Selecting the right K is one of the most important and challenging decisions in clustering.

The first step of K-Means is to randomly place K centroids somewhere in the data space, usually by picking K real data points at random as starting positions. All subsequent steps — assigning each point to the nearest centroid and then moving centroids — build on this initial random placement. A poor random start can lead to poor final clusters, which is why smarter initialization methods like K-Means++ were invented.

The most common measure is Euclidean distance — the straight-line distance between two points, just like measuring with a ruler. In 2D it uses the Pythagorean theorem: distance = √((x2−x1)² + (y2−y1)²). Points with a shorter distance are considered more similar and are more likely to end up in the same cluster.

import math
p1, p2 = (1, 2), (4, 6)
dist = math.sqrt((p2[0]-p1[0])**2 + (p2[1]-p1[1])**2)
print(dist)  # 5.0

In each iteration, every data point is first assigned to the nearest centroid based on distance. Then each centroid is moved to the mean (average) position of all the points assigned to it. This assign-then-move cycle repeats until the centroids stop changing, which is called convergence.

K-Means stops when the centroids no longer move (or move by less than a tiny threshold between rounds), meaning the cluster assignments have stabilized — this is called convergence. It may also stop after reaching a preset maximum number of iterations to avoid running forever. In practice, K-Means usually converges in tens to hundreds of iterations.

The Elbow Method helps you choose the best value of K by plotting inertia (total within-cluster squared distance) against different values of K. As K grows, inertia shrinks, but after the right K the improvement becomes very small — the graph bends like an elbow. You pick the K at this elbow point as a good balance between simplicity and cluster quality.

Features with large numerical ranges (like annual salary in the tens of thousands) can dominate distance calculations and overshadow features with small ranges (like age 0–100), making clustering unfair. Scaling brings all features to a similar range so each one contributes equally. Without scaling, K-Means clusters may simply reflect differences in feature magnitude rather than true similarity.

An outlier is a data point that is very different from all others — for example, a customer who spends 100× more than everyone else. In K-Means, outliers can drag centroids far away from the true center of a cluster, distorting all nearby groups. It is good practice to detect and handle outliers before running K-Means to get meaningful results.

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) groups points that are packed closely together in dense regions and automatically labels isolated points as noise (outliers). Unlike K-Means, you do not need to specify the number of clusters beforehand. DBSCAN can also find clusters of any shape — not just round blobs — making it more flexible for real-world data.

Hierarchical clustering builds a tree of clusters by repeatedly merging the two closest clusters (agglomerative, bottom-up) or by repeatedly splitting one large cluster (divisive, top-down). The result is a tree diagram called a dendrogram that shows every merge or split. One major advantage is that you can cut the tree at any level to get any number of clusters without rerunning the algorithm.

A dendrogram is a tree-shaped diagram produced by hierarchical clustering showing how data points are progressively merged into clusters step by step. The height at which two branches merge shows how different those groups were when they joined — higher merges mean the groups were more dissimilar. By drawing a horizontal line across the dendrogram at a chosen height, you decide how many final clusters to keep.

The silhouette score measures how well each data point fits inside its own cluster compared to neighboring clusters. Scores range from −1 to +1: near +1 means the point is well-placed inside its cluster, 0 means it sits on the border between two clusters, and a negative score suggests it may belong to the wrong cluster. Averaging silhouette scores across all points gives an overall quality rating for the entire clustering.

Inertia is the total sum of squared distances between every data point and its assigned centroid — a measure of how tightly packed the clusters are. Lower inertia means points are closer to their centroids, indicating more compact, well-defined clusters. However, inertia always decreases as K increases, so you cannot just minimize inertia to choose K — you need the Elbow Method to find a sensible stopping point.

K-Means assigns each point to its nearest centroid and then computes the mean, which works perfectly for round, evenly-sized blobs of data. For elongated, crescent-shaped, or irregularly shaped clusters, the centroid can fall outside the actual cluster, causing wrong assignments and splitting real groups in two. Algorithms like DBSCAN handle non-spherical shapes much better because they rely on density rather than distance to a fixed center.

A digital image contains millions of pixels, each with a color value drawn from up to 16 million possible RGB combinations. K-Means can cluster all pixel colors into just K representative colors (for example, K=16), and then replace every pixel's original color with the nearest centroid color. The resulting image uses far fewer distinct colors and takes much less storage while still looking recognizable — a technique called color quantization.

Customer segmentation is the classic example: a business groups its customers into segments (for example, 'bargain hunters,' 'luxury buyers,' 'occasional shoppers') based on purchase behavior, without anyone manually labeling each customer. This lets companies design targeted marketing for each group automatically. The computer finds the segments on its own purely by detecting patterns in spending data.

Two data points end up in the same cluster because they are more similar to each other — measured by distance — than they are to points in other clusters. The shorter the distance between two points, the more likely they will be grouped together. It is similar to how students with the same interests naturally form friend groups at school.

Manhattan distance measures the total distance traveled along grid-like paths — only horizontal and vertical moves allowed — like navigating city blocks. It is calculated as the sum of absolute differences: |x2−x1| + |y2−y1|, versus Euclidean distance, which is the straight diagonal line. Manhattan distance is useful when movement must follow a grid, or in high-dimensional data where straight-line distance can become misleading.

p1, p2 = (1, 2), (4, 6)
manhattan = abs(p2[0]-p1[0]) + abs(p2[1]-p1[1])
print(manhattan)  # 7

K-Means starts by randomly placing its K centroids, and different random starting positions can send the algorithm down different paths to different local solutions (local minima). The algorithm is not guaranteed to find the globally best clustering — it only finds the best solution from wherever it starts. This is why it is standard practice to run K-Means multiple times with different random seeds and pick the run with the lowest inertia.

K-Means++ is a smarter initialization strategy that spreads the initial centroids far apart from each other. It picks the first centroid randomly, and each subsequent centroid is chosen with a probability proportional to its squared distance from the already-chosen centroids — so far-away points are preferred. This greatly reduces the chance of a bad random start and typically leads to better clusters and faster convergence.

A core point is a data point that has at least a minimum required number of neighbors (minPts) within a specified radius (epsilon, ε). Core points are the foundation of dense clusters — any core point and all points reachable from it form a cluster. Points within ε of a core point but without enough neighbors of their own are border points, while isolated points with no nearby core are labeled noise (outliers).

Classification is supervised learning — you train a model on labeled examples and it learns to predict a known category for new data (like 'spam' or 'not spam'). Clustering is unsupervised — there are no labels, and the algorithm discovers hidden groups on its own. Think of classification as sorting mail using a system you already understand, while clustering is sorting an unknown pile of letters by spotting which ones look alike.

Streaming apps can cluster songs by audio features such as tempo, energy, key, and loudness into groups of similar-sounding tracks. When you enjoy a song, the app recommends others from the same cluster that are likely to match your taste. This is how the 'you might also like…' feature works — it is powered by unsupervised clustering running behind the scenes.

Biologists apply clustering to group genes that behave similarly across different experimental conditions or cell types, which can reveal genes that work together in the same biological pathway. For example, if several genes are all switched on at the same time during an infection, they may cluster together and hint at a shared function. This technique, called gene expression clustering, has helped scientists discover new disease mechanisms.

When clustering groups people by data like neighborhood, purchases, or online behavior, it can accidentally encode and reinforce existing biases — for example, showing higher loan interest rates or withholding job ads from certain demographic groups. The data features chosen for clustering may act as proxies for protected characteristics like race or gender, leading to discriminatory outcomes even when no one intended harm. It is essential to examine whether your clustering features could unfairly reflect or amplify societal inequalities.

Inertia keeps decreasing as K grows because more clusters means each cluster is smaller and its points sit closer to their centroid. However, after the ideal K, each extra cluster adds very little real structure — you are essentially splitting natural groups unnecessarily. This is exactly why the Elbow Method looks for the 'bend' where inertia improvement flattens out, signaling diminishing returns.

When clusters overlap, data points near the boundary could reasonably belong to either cluster, making any hard assignment ambiguous and likely wrong. Standard K-Means forces every point into exactly one cluster, which misrepresents this fuzzy boundary. Soft or fuzzy clustering methods — such as Fuzzy C-Means or Gaussian Mixture Models — solve this by giving each point a probability of belonging to each cluster rather than a single hard label.

Mini-Batch K-Means is a faster variant of K-Means that updates centroids using small random subsets (mini-batches) of the data at each step rather than the full dataset. This makes it dramatically faster on very large datasets while producing cluster quality nearly as good as standard K-Means. It is especially valuable when the entire dataset is too large to fit in the computer's memory at once.

Irrelevant features add noise to the distance calculations, making unrelated data points seem similar or truly similar points seem different. For example, if you cluster students by exam scores but accidentally include their shoe size, the shoe size adds meaningless distance that distorts the groupings. Choosing only meaningful features — or reducing dimensions first — leads to much cleaner, more useful clusters.

You first cluster the normal (typical) data so the algorithm learns what regular patterns look like. Then any new data point that falls very far from all cluster centroids is flagged as a potential anomaly — it does not fit any normal group. In fraud detection, most bank transactions cluster into expected spending patterns, and a transaction that lands far from every cluster is automatically flagged as suspicious for human review.

This code imports K-Means from scikit-learn, creates a model configured to find 3 clusters, fits it to the data, and then prints the cluster label (0, 1, or 2) assigned to each data point. To find 5 clusters instead, you simply change `n_clusters=3` to `n_clusters=5`. The `random_state=42` ensures the same random starting positions are used every run, making the results reproducible.

from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=3, random_state=42)
kmeans.fit(data)
print(kmeans.labels_)