Classification: sorting into groups

Classification is the task of assigning input data to one of several predefined categories or classes. For example, sorting emails into 'spam' or 'not spam' is a classic classification problem. A computer learns these categories by studying many labeled examples during a training phase.

A label (also called a class) is the category we want the computer to predict for each example. For instance, in a fruit classifier, 'apple,' 'banana,' and 'mango' are labels. Every training example must have a label attached so the machine knows what it is learning to predict.

Supervised learning is when a machine learns from examples that already have correct answers attached — those answers are called labels. Think of a teacher marking exam papers: the model studies many labeled examples and learns to predict labels for new, unseen data. Classification is one of the two main types of supervised learning (the other being regression).

A feature is a measurable property of the data that the classifier uses to make its decision. For example, when classifying animals, features might include number of legs, whether it has fur, or its weight. The more informative and relevant the features are, the more accurately the classifier can sort examples into the correct groups.

Binary classification means sorting every data point into exactly one of two possible categories, such as 'yes/no,' 'pass/fail,' or 'spam/not spam.' The model outputs one of only two possible labels for each input. Most beginner classification problems are binary, and many real-world tasks — like fraud detection — are framed this way.

A training dataset is the collection of labeled examples that a classification model learns from. The model studies the patterns in these examples — the relationship between features and labels — and adjusts its internal rules to make correct predictions. A good training dataset should be large, diverse, and representative of the real-world situations the model will face.

We use a separate test dataset to measure how well the model performs on data it has never seen before. If we tested on the same data used for training, the model might simply have memorized the answers rather than truly learning the underlying pattern — giving a falsely high score. The test set provides an honest measure of real-world performance.

A decision tree is a classification algorithm that makes predictions by asking a series of yes/no questions about the input features, branching left or right based on each answer. Imagine a flowchart: 'Does it have feathers? Yes → Does it fly? Yes → Bird.' It keeps splitting until it reaches a leaf node that holds the final class label. Decision trees are popular because they are easy to visualize and explain.

Accuracy is the fraction of total predictions the model gets correct (correct predictions divided by total predictions). It is misleading when classes are heavily imbalanced — for example, if 99% of transactions are legitimate, a model that always predicts 'legitimate' achieves 99% accuracy while being completely useless at catching fraud. In such cases, metrics like precision, recall, and F1-score give a more honest picture.

KNN classifies a new data point by finding its K closest neighbors in the training data (measured by distance, such as straight-line distance) and taking a majority vote of their labels. For example, if K=3 and two neighbors are labeled 'dog' and one is 'cat,' the model predicts 'dog.' KNN is intuitive because it essentially asks: 'What do similar examples around me look like?'

Overfitting happens when a model learns the training data so thoroughly — including random noise and quirks — that it fails to generalize to new examples. It's like a student who memorizes every past exam question word-for-word but cannot solve any new problem. You can detect overfitting when training accuracy is very high but test (or validation) accuracy is significantly lower.

Underfitting happens when a model is too simple to capture the important patterns in the data, resulting in poor accuracy on both training and test sets. Imagine trying to describe all animals with just one rule — 'it has a body' — which tells us almost nothing useful. Underfitting is usually caused by using too few features, too simple an algorithm, or stopping training too early.

A confusion matrix is a grid (table) that shows how many times the model predicted each class versus what the actual class was. The rows typically represent actual classes and the columns represent predicted classes. It reveals not just overall accuracy but also which specific classes the model tends to confuse with each other — which is far more useful for diagnosing and improving a classifier.

Multiclass classification involves sorting data into three or more distinct categories — for example, classifying a news article as 'sports,' 'politics,' 'technology,' or 'entertainment.' Binary classification has exactly two classes, while multiclass has many. Many real-world problems are multiclass, and some algorithms need special adaptations to handle more than two categories.

Precision measures what fraction of the model's positive predictions are actually correct: true positives divided by all predicted positives. For a spam filter, precision asks: 'Of all the emails I marked as spam, how many truly were spam?' High precision is especially important when the cost of a false alarm is high — for example, you don't want a spam filter to accidentally delete important emails by mistakenly labeling them spam.

Recall measures what fraction of all actual positives the model correctly identified: true positives divided by all actual positives. For a disease detector, recall asks: 'Of all patients who truly have the disease, how many did the model catch?' High recall is critical when missing a positive case is very dangerous — such as missing a cancer diagnosis — even if it means some false alarms.

Cross-validation (commonly k-fold) splits the training data into k equal parts, then trains and evaluates the model k times — each time holding out a different part as the validation set and using the rest for training. Averaging performance across all k runs gives a more reliable estimate than a single split, because it reduces the luck of which particular examples end up in the validation set. A common choice is k=5 or k=10.

Class imbalance means one class has far more training examples than another — for instance, 950 'healthy' patients and only 50 'sick' ones. A naive model might always predict 'healthy' and still achieve 95% accuracy while being completely useless for detecting illness. Solutions include oversampling the minority class (e.g., SMOTE), undersampling the majority class, or using metrics like F1-score instead of accuracy.

A naive Bayes classifier uses Bayes' theorem to calculate the probability of each class given the input features, then predicts the class with the highest probability. It is called 'naive' because it assumes all features are completely independent of each other — an assumption that is almost never true in practice. Despite this simplification, naive Bayes works surprisingly well for tasks like spam filtering and text categorization.

Feature engineering is the process of creating, transforming, or selecting the most informative input features from raw data before training a classifier. For example, extracting 'day of week' from a raw date, or combining height and weight into BMI. Well-crafted features often improve accuracy more than switching to a fancier algorithm, because the model can only learn from the information you give it.

An ensemble method improves classification by combining predictions from many individual models rather than relying on one. Random Forest builds hundreds of decision trees, each trained on a random subset of the training data and a random subset of features, then takes a majority vote of all trees' predictions. This reduces overfitting and usually achieves higher accuracy than any single tree.

Two examples: (1) A doctor's scan classifier that predicts whether a tumor is 'benign' or 'malignant,' and (2) a digit recognizer that classifies a handwritten image as '0' through '9.' Both are classification problems because the output is a discrete category chosen from a fixed set of labels — not a continuous number like a price or temperature. Classification is used any time the goal is to assign an item to a group.

During training, the model repeatedly looks at labeled examples, makes a prediction for each one, and then measures how wrong it was using a loss function. It then adjusts its internal parameters (weights or rules) to reduce that error, and repeats this process over many examples. After enough iterations, the model has tuned itself to capture the patterns that connect features to the correct labels.

Classification predicts a discrete category — such as 'cat,' 'dog,' or 'bird' — while regression predicts a continuous numerical value, such as the price of a house or tomorrow's temperature. Both are supervised learning tasks that learn from labeled examples, but they use different output types, loss functions, and evaluation metrics. Choosing the right task type is the first step in any ML project.

If a classifier is trained on biased data, it can learn and amplify unfair patterns. For example, a face recognition system trained mostly on light-skinned faces may perform poorly on darker-skinned faces, leading to misidentification. When such systems are used in hiring, lending, or policing, they can discriminate against certain groups. It is crucial to audit training data for bias and to measure model performance separately across different demographic groups.

Logistic regression calculates the probability that an input belongs to the positive class (e.g., P(spam) = 0.87) using an S-shaped probability curve called the sigmoid function. It then applies a threshold — typically 0.5 — to assign a final label: if P ≥ 0.5, predict positive; otherwise, predict negative. It works best for binary classification, is fast to train, and produces easily interpretable probability scores.

A hyperparameter is a setting you choose before training — it controls how the model learns but is not learned from the data itself. In KNN, the number of neighbors K is a hyperparameter. If K is too small (e.g., K=1), the model is overly sensitive to noise and overfits; if K is too large, it averages over too much of the data and may underfit. The best K is found by testing different values on a validation set.

A validation set is used during model development to tune hyperparameters and compare different classifiers — you are allowed to look at validation scores and make changes based on them. The test set is held completely aside and used only once, at the very end, to report the final honest performance. Think of the validation set as a practice exam and the test set as the real final exam that you only sit once.

For multiclass classifiers, compute precision, recall, and F1-score separately for each class, then summarize with a macro or weighted average. A confusion matrix also reveals which specific classes the model is confusing with each other — for example, perhaps it often mistakes 'cat' for 'dog' but rarely mistakes 'car' for anything. This multi-metric approach is especially important when some classes have far fewer examples than others.

Neural networks are classification models made of layers of interconnected nodes (neurons) that progressively transform input features into higher-level representations. The final layer outputs a probability score for each class, and the class with the highest score is selected as the prediction. Deep neural networks are especially powerful at classifying images, speech, and text because they can automatically discover complex features without manual feature engineering.

Training a classifier means showing it many labeled examples — data points where the correct answer is already known — so it can figure out the patterns that connect input features to the right labels. Think of it like a student who studies many solved practice problems: the more good examples the model sees, the better it gets at predicting the correct label for brand-new examples it has never seen before.

With K=1, the classifier assigns the label of the single nearest neighbor, making it extremely sensitive to noise and outliers — this leads to overfitting. With a very large K (say K=100 in a small dataset), the model considers so many neighbors that local structure is lost and the model underfits. The best K is found by trying several values on a validation set and picking the one with the highest validation accuracy.

from sklearn.neighbors import KNeighborsClassifier
# Try different K values on validation data
for k in [1, 5, 11, 21]:
    clf = KNeighborsClassifier(n_neighbors=k)
    clf.fit(X_train, y_train)
    print(f"K={k}: val accuracy = {clf.score(X_val, y_val):.2f}")

The F1-score is the harmonic mean of precision and recall: F1 = 2 × (Precision × Recall) / (Precision + Recall). It balances both metrics into a single number, making it especially useful when classes are imbalanced or when both false positives and false negatives carry real costs. A perfect F1-score is 1.0, while a completely useless classifier scores 0.0.

In scikit-learn, the `.fit(X_train, y_train)` method trains a classifier, where `X_train` is a table of feature values and `y_train` is a list of the correct labels. This same method signature works for almost every classifier in scikit-learn — whether it is a decision tree, KNN, or random forest — making it very consistent and beginner-friendly. After fitting, call `.predict(X_test)` to classify new examples.

from sklearn.tree import DecisionTreeClassifier

# Features: [height_cm, weight_kg]  Labels: 0=child, 1=adult
X_train = [[120, 25], [160, 55], [170, 70], [110, 20]]
y_train = [0, 1, 1, 0]

clf = DecisionTreeClassifier()
clf.fit(X_train, y_train)          # <-- this line trains the model
print(clf.predict([[150, 45]]))    # Output: [1]  (adult)