Supervised learning algorithms

Supervised learning trains a model on labeled data, where each input example is paired with a known correct output (target), so the model learns to map inputs to outputs. Unsupervised learning works with unlabeled data and instead discovers hidden structure such as clusters or low-dimensional representations. The presence of ground-truth labels is the defining feature of supervised learning.

Classification predicts a discrete category or class label (e.g., spam vs. not spam), while regression predicts a continuous numerical value (e.g., house price). Both are supervised because they use labeled training data, but the nature of the target variable differs. The choice of loss function and evaluation metric depends on which type of task it is.

Linear regression assumes the target is approximately a linear (weighted) combination of the input features plus an intercept, i.e., y ≈ w·x + b. It fits coefficients so that this straight-line (or hyperplane) relationship best matches the data. If the true relationship is strongly nonlinear, a plain linear model will underfit.

MSE averages the squared differences between predicted and actual values: MSE = (1/n) Σ(yᵢ − ŷᵢ)². Squaring penalizes larger errors more heavily and keeps the loss differentiable, which makes gradient-based optimization convenient. It is widely used because minimizing it corresponds to fitting the conditional mean of the target.

import numpy as np
mse = np.mean((y_true - y_pred) ** 2)

Logistic regression outputs a probability between 0 and 1 by passing a linear combination of features through the sigmoid function, and a threshold (often 0.5) converts that probability into a class label. So although it fits a linear model internally, its purpose is to assign discrete class labels. The word 'regression' refers to the underlying linear model, not the task.

import numpy as np
p = 1 / (1 + np.exp(-(w @ x + b)))
label = 1 if p >= 0.5 else 0

The sigmoid function, σ(z) = 1/(1+e^(−z)), squashes any real-valued input into the range (0, 1), allowing the model's linear output to be interpreted as a probability. As z grows large and positive the output approaches 1, and as z grows large and negative it approaches 0. This S-shaped mapping is what turns a linear score into a usable class probability.

Overfitting happens when a model learns noise and idiosyncrasies of the training data instead of the general pattern, so it performs very well on training data but poorly on unseen test data. You recognize it by a large gap between high training accuracy and low validation/test accuracy. It is often caused by excessive model complexity relative to the amount of data.

Underfitting occurs when a model is too simple to capture the underlying pattern, leading to high error on both the training and test sets. Common causes include using a model with too little capacity (e.g., a linear model for nonlinear data), too few features, or excessive regularization. The fix is usually a more expressive model or richer features.

Bias is error from overly simplistic assumptions (causing underfitting), while variance is error from excessive sensitivity to the training data (causing overfitting). Increasing model complexity typically lowers bias but raises variance, and vice versa. The goal is to find a balance that minimizes total expected error on unseen data.

The data is split so the model learns from the training set and is then evaluated on a separate test set it has never seen, giving an honest estimate of how it will generalize to new data. Evaluating on the same data used for training would give an overly optimistic score. The test set must remain untouched during training and tuning.

from sklearn.model_selection import train_test_split
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

A decision tree routes an example from the root down through a series of feature-based yes/no questions until it reaches a leaf node. The leaf's stored value (a majority class for classification or an average for regression) becomes the prediction. Each internal split is chosen during training to best separate the data.

They commonly use Gini impurity or entropy (information gain), both of which measure how mixed the class labels are within a node. A split is chosen to maximize the reduction in impurity, producing child nodes that are more homogeneous than the parent. Lower impurity means the node is closer to containing a single class.

A deep, unconstrained tree can keep splitting until each leaf contains very few (or one) training examples, effectively memorizing the data including its noise. This is controlled by limiting maximum depth, requiring a minimum number of samples per leaf or split, or pruning the tree after growth. These constraints trade a little training accuracy for better generalization.

from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier(max_depth=4, min_samples_leaf=10)

A Random Forest trains many decision trees on bootstrapped samples of the data and random subsets of features, then averages or votes across their predictions. This ensemble reduces variance and the overfitting that plagues a single deep tree, usually giving more stable and accurate results. The diversity among trees is key to the improvement.

from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(n_estimators=200)

Bagging trains multiple models on different bootstrap samples (random samples drawn with replacement) of the training data and combines their predictions by voting or averaging. Because each model sees a slightly different dataset, their errors are partly uncorrelated, so averaging reduces overall variance. Random Forests are a classic example of bagging applied to decision trees.

k-NN finds the k training examples closest to the new point (by a distance metric such as Euclidean distance) and assigns the majority class among those neighbors. It is a 'lazy' learner because it does no real training—it simply stores the data and computes distances at prediction time. The choice of k controls how smooth the decision boundary is.

from sklearn.neighbors import KNeighborsClassifier
clf = KNeighborsClassifier(n_neighbors=5)

Distance-based methods compute distances across all features, so a feature with a large numeric range (e.g., salary in thousands) can dominate one with a small range (e.g., age), distorting which points count as 'near.' Scaling (e.g., standardization or min-max normalization) puts features on comparable scales so each contributes fairly. Without it, the model may effectively ignore small-range features.

from sklearn.preprocessing import StandardScaler
X_scaled = StandardScaler().fit_transform(X)

k is the number of nearest neighbors that vote on the prediction. A very small k (like 1) makes the model sensitive to noise and can overfit with a jagged boundary, while a very large k oversmooths and may underfit by ignoring local structure. Choosing k well (often via cross-validation) balances these effects.

An SVM finds the hyperplane that separates the classes with the largest possible margin—the widest gap between the boundary and the nearest data points of each class. Those nearest points are the support vectors and they alone define the boundary. Maximizing the margin tends to improve generalization to new data.

from sklearn.svm import SVC
clf = SVC(kernel='rbf', C=1.0)

The kernel trick lets an SVM compute relationships between data points as if they were mapped into a much higher-dimensional space, without ever explicitly computing those coordinates. This enables the SVM to find nonlinear decision boundaries (e.g., with an RBF or polynomial kernel) while keeping computation efficient. It effectively makes linearly-inseparable data separable in the transformed space.

With class imbalance (e.g., 99% negatives), a model can achieve 99% accuracy by always predicting the majority class while completely failing to detect the rare positive class. Accuracy hides this failure because it weights all examples equally. Metrics like precision, recall, F1-score, or AUC give a clearer picture of minority-class performance.

Precision is the fraction of predicted positives that are actually positive: TP/(TP+FP). Recall (sensitivity) is the fraction of actual positives the model correctly identifies: TP/(TP+FN). Precision answers 'when we predict positive, how often are we right?' while recall answers 'of all true positives, how many did we catch?'

The F1-score is the harmonic mean of precision and recall: F1 = 2·(precision·recall)/(precision+recall). It is preferred when classes are imbalanced or when both false positives and false negatives matter, because it rewards a balance between precision and recall. A high F1 requires the model to be good at both, not just one.

A confusion matrix tabulates predictions against actual labels, showing counts of true positives, true negatives, false positives, and false negatives. This breakdown reveals not just how often the model is wrong but in what way (which classes get confused). Many metrics like precision, recall, and accuracy are derived directly from its entries.

from sklearn.metrics import confusion_matrix
confusion_matrix(y_true, y_pred)

Gradient descent is an iterative optimization method that updates model parameters in the direction opposite to the gradient of the loss function, gradually moving toward a minimum. At each step, parameters are adjusted by the gradient scaled by a learning rate. Repeated over many iterations, this drives the loss down until convergence.

w = w - learning_rate * gradient

If the learning rate is too large, the updates may overshoot the minimum and cause the loss to oscillate or diverge. If it is too small, training converges very slowly and may stall before reaching a good solution. A well-chosen learning rate lets the model converge steadily and efficiently.

Regularization adds a penalty on large parameter values to the loss function, discouraging overly complex models and thereby reducing overfitting. L2 (ridge) shrinks weights smoothly toward zero, while L1 (lasso) can drive some weights exactly to zero, performing feature selection. The strength is controlled by a hyperparameter that balances fit against simplicity.

from sklearn.linear_model import Ridge, Lasso
model = Ridge(alpha=1.0)

L1 regularization tends to produce sparse models by driving some coefficients exactly to zero, effectively selecting a subset of features. L2 regularization shrinks all coefficients smoothly toward zero but rarely makes them exactly zero, keeping all features but with smaller weights. So L1 is useful for feature selection, while L2 is good for handling correlated features and general shrinkage.

K-fold cross-validation splits the data into k equal parts, trains the model k times each using k−1 folds for training and the remaining fold for validation, then averages the results. This gives a more reliable estimate of generalization than a single train/test split and uses the data more efficiently. It is especially valuable when the dataset is small.

from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X, y, cv=5)

Parameters (like the weights in linear regression) are learned automatically from the training data during fitting. Hyperparameters (like the learning rate, tree depth, or k in k-NN) are set before training and control how learning proceeds. Hyperparameters are typically tuned using a validation set or cross-validation rather than learned directly.

Naive Bayes uses Bayes' theorem to compute the probability of each class given the features and predicts the class with the highest posterior probability. The 'naive' assumption is that all features are conditionally independent given the class, which simplifies the math greatly. Despite this often-unrealistic assumption, it works surprisingly well, especially for text classification.

from sklearn.naive_bayes import GaussianNB
clf = GaussianNB()

One-hot encoding converts a categorical variable into several binary columns, one per category, where exactly one column is 1 for each row. It is used because most algorithms require numeric input and because it avoids implying a false ordinal relationship between categories (e.g., red < green < blue). The trade-off is that it increases the number of features.

import pandas as pd
encoded = pd.get_dummies(df['color'])

Boosting builds an ensemble of weak learners sequentially, where each new model focuses on correcting the errors made by the combined previous models. Gradient boosting does this by fitting each new tree to the residual errors (gradients) of the current ensemble. The models are combined into a strong predictor, typically reducing bias.

from sklearn.ensemble import GradientBoostingClassifier
clf = GradientBoostingClassifier()

Bagging trains many models in parallel on independent bootstrap samples and averages them, mainly reducing variance. Boosting trains models sequentially, each correcting the previous models' mistakes, mainly reducing bias. As a result boosting can achieve very high accuracy but is more sensitive to noise and overfitting if not regularized.

As the number of features (dimensions) grows, the volume of the feature space increases so fast that data points become sparse and distances between points become less meaningful. This hurts distance-based methods like k-NN and increases the risk of overfitting because more data is needed to cover the space. Dimensionality reduction or feature selection can help mitigate it.

The ROC curve plots the true positive rate against the false positive rate across all classification thresholds, showing the trade-off as the threshold changes. The area under this curve (AUC) summarizes overall ranking quality: 1.0 is perfect, 0.5 is no better than random guessing. AUC is threshold-independent, making it useful for comparing classifiers.

from sklearn.metrics import roc_auc_score
auc = roc_auc_score(y_true, y_scores)

If you tune hyperparameters using the test set, the test performance no longer reflects true generalization because the model has indirectly 'seen' that data through repeated tuning—a form of leakage. A separate validation set (or cross-validation) is used for tuning so the test set remains a clean, final, unbiased measure of performance. This keeps the evaluation honest.

Data leakage occurs when information that would not be available at prediction time (such as test data statistics or a feature derived from the target) influences model training. It produces overly optimistic validation scores that collapse when the model faces real new data. A common cause is fitting preprocessing steps on the whole dataset before splitting.

Raising the polynomial degree lets the model fit increasingly wiggly curves, which lowers training error but greatly increases the risk of overfitting and high variance. A very high-degree polynomial can pass through every training point yet generalize terribly to new data. Regularization or choosing a modest degree via validation helps control this.

from sklearn.preprocessing import PolynomialFeatures
Xp = PolynomialFeatures(degree=3).fit_transform(X)

Stratified sampling preserves the original proportion of each class in both the training and test sets, so each split is representative of the full dataset. This is especially important for imbalanced data, where a random split might leave very few (or zero) minority examples in one set. It leads to more reliable training and evaluation.

train_test_split(X, y, test_size=0.2, stratify=y)