Machine Learning Fundamentals

Supervised learning trains a model on labeled data, where each input has a known target output, so the model learns to predict that output. Unsupervised learning works with unlabeled data and tries to find hidden structure or patterns, such as clusters or lower-dimensional representations. In short, supervised learning predicts known labels while unsupervised learning discovers structure.

We train the model on the training set and evaluate it on the separate test set to estimate how well it generalizes to new, unseen data. If we evaluated on the same data used for training, the score would be overly optimistic because the model may have memorized those examples. The test set acts as a proxy for real-world performance.

Overfitting happens when a model learns the training data too closely, including its noise and random quirks, so it performs very well on training data but poorly on new data. It usually indicates the model is too complex relative to the amount of data. The gap between high training accuracy and low test accuracy is a classic sign.

Underfitting occurs when a model is too simple to capture the underlying patterns in the data, so it performs poorly on both training and test sets. It is often caused by an insufficiently flexible model, too few features, or excessive regularization. The fix is usually a more complex model or better features.

Classification is a supervised learning task because it is trained on labeled examples. It predicts a discrete category or class label for each input, such as 'spam' versus 'not spam' or the species of a flower. The output is one of a finite set of classes rather than a continuous number.

Both are supervised tasks, but regression predicts a continuous numerical value (like house price or temperature), while classification predicts a discrete category. For example, predicting the exact price of a house is regression, whereas predicting whether it is 'cheap' or 'expensive' is classification.

A feature is an individual measurable property or input variable used to describe each example, often shown as a column in a table. For instance, in predicting house prices, features could be area, number of bedrooms, and location. The model uses these features to make predictions about the target.

The label (or target) is the correct output value that the model is trying to predict for each training example. In a spam detector, the label might be 'spam' or 'not spam'; in price prediction, it is the actual price. The model learns the relationship between features and labels during training.

Clustering is an unsupervised technique that groups data points so that points within the same group are more similar to each other than to points in other groups. K-means specifically partitions data into k clusters by minimizing the distance of points to their cluster's center (centroid). It is used for tasks like customer segmentation where no labels exist.

The parameter k is the number of clusters you want the algorithm to find, and it must be chosen before running k-means. The algorithm then assigns every data point to one of these k groups and computes k centroids. Choosing a good k is important and is often guided by methods like the elbow method.

A loss function measures how far the model's predictions are from the true target values, giving a single number that quantifies error. Training works by adjusting the model's parameters to minimize this loss. Common examples are mean squared error for regression and cross-entropy for classification.

Gradient descent is an optimization algorithm that iteratively adjusts model parameters to minimize the loss function. It computes the gradient (slope) of the loss with respect to each parameter and takes small steps in the opposite direction of the gradient. Repeating this gradually moves the parameters toward values that produce lower error.

The learning rate controls the size of the steps taken when updating parameters during gradient descent. If it is too large, the algorithm may overshoot the minimum and fail to converge; if it is too small, training becomes very slow. Choosing a suitable learning rate is an important part of tuning a model.

Parameters are values learned by the model from the data during training, such as the weights in linear regression. Hyperparameters are settings chosen before training that control the learning process, such as the learning rate or the number of clusters k. We tune hyperparameters but the model itself learns its parameters.

The bias-variance tradeoff describes the balance between two sources of error: bias (error from overly simple assumptions, causing underfitting) and variance (error from sensitivity to small fluctuations in training data, causing overfitting). Reducing one often increases the other, so good models find a balance that minimizes total error on unseen data.

A validation set is used during model development to tune hyperparameters and compare models, while the test set is reserved for a final, unbiased performance estimate. Using a validation set prevents you from indirectly fitting the test set through repeated tuning. A common split is training, validation, and test.

K-fold cross-validation splits the data into k equal parts, then trains on k-1 parts and validates on the remaining part, repeating this k times so each part serves as validation once. The results are averaged to give a more reliable estimate of model performance than a single split. It is especially useful when data is limited.

A false positive occurs when the model predicts the positive class but the true label is actually negative. For example, a spam filter marking a legitimate email as spam is a false positive. It is also called a Type I error.

Accuracy is the fraction of predictions the model gets correct, calculated as the number of correct predictions divided by the total number of predictions. While intuitive, it can be misleading on imbalanced datasets where one class dominates. In such cases, metrics like precision, recall, or F1 score are more informative.

On an imbalanced dataset, one class greatly outnumbers the other, so a model can achieve high accuracy simply by always predicting the majority class. For example, if 99% of emails are not spam, a model that labels everything 'not spam' is 99% accurate but useless for catching spam. Metrics like precision, recall, and F1 give a clearer picture.

Precision measures, of all the items predicted positive, how many were actually positive, reflecting how trustworthy positive predictions are. Recall measures, of all the actual positives, how many the model correctly identified, reflecting how many it missed. There is often a tradeoff between them depending on the chosen decision threshold.

The F1 score is the harmonic mean of precision and recall, combining both into a single number that is high only when both are reasonably high. It is preferred over accuracy when classes are imbalanced or when both false positives and false negatives matter. The harmonic mean penalizes extreme imbalance between precision and recall.

Feature scaling puts numerical features onto a comparable range so that features with naturally large values do not dominate those with small values. This is important for distance-based algorithms like k-nearest neighbors and k-means, and it helps gradient descent converge faster. Common methods are min-max normalization and standardization (z-score).

KNN finds the k training examples closest to the new point (usually by Euclidean distance) and assigns the class that is most common among those neighbors. It is a 'lazy' learner because it stores the training data and does the work at prediction time rather than building an explicit model. The choice of k and the distance metric strongly affect results.

A decision tree makes predictions by asking a sequence of yes/no questions about the features, splitting the data at each node to separate the classes or reduce error. Each path from the root to a leaf represents a decision rule, and the leaf gives the prediction. Trees are easy to interpret but can overfit if grown too deep.

A random forest is an ensemble of many decision trees trained on random subsets of the data and features, whose predictions are combined by voting (classification) or averaging (regression). By averaging many diverse trees, it reduces the high variance and overfitting of a single deep tree. This usually gives more accurate and stable predictions.

Logistic regression is used for classification, not regression, despite its name. It models the probability that an input belongs to a particular class by passing a linear combination of features through the sigmoid (logistic) function, which outputs a value between 0 and 1. A threshold (often 0.5) then converts this probability into a class label.

The sigmoid function maps any real number to a value between 0 and 1, following an S-shaped curve. This makes its output interpretable as a probability, which is why it is used in logistic regression and as a neuron activation. Inputs near zero give outputs near 0.5, while large positive or negative inputs saturate near 1 or 0.

import math
def sigmoid(x):
    return 1 / (1 + math.exp(-x))
print(sigmoid(0))  # 0.5

Simple linear regression models the relationship between one feature x and the target y as y = mx + b, where m is the slope (weight) and b is the intercept (bias). Training finds the values of m and b that best fit the data, usually by minimizing the mean squared error. The result is a straight line predicting y from x.

Mean squared error is the average of the squared differences between predicted values and actual values. Squaring makes all errors positive and penalizes larger errors more heavily, and a lower MSE indicates a better fit. It is one of the most common loss functions for regression.

Regularization adds a penalty term to the loss function that discourages overly large model weights, keeping the model simpler. L2 regularization penalizes the sum of squared weights, shrinking them toward zero and reducing sensitivity to noise. This typically improves generalization to unseen data.

Dimensionality reduction reduces the number of features while retaining as much important information as possible. Principal Component Analysis (PCA) does this by finding new axes (principal components) along which the data varies the most and projecting the data onto them. This helps with visualization, speeds up training, and can reduce noise and overfitting.

Two features are highly correlated when they tend to change together in a predictable way, so knowing one gives strong information about the other. Positive correlation means they increase together, while negative correlation means one increases as the other decreases. Highly correlated features can be redundant and may cause instability in some models.

Two variables can be correlated without one causing the other, perhaps due to coincidence or a hidden third factor that influences both. Treating correlation as causation can lead to wrong conclusions and bad decisions. Establishing causation generally requires controlled experiments or careful causal analysis, not just observed correlation.

The mean is the arithmetic average of all values, found by summing them and dividing by the count. The median is the middle value when the data is sorted, splitting it into two equal halves. The median is more robust to outliers, so it is often preferred for skewed data like incomes.

Standard deviation measures how spread out the values are around the mean. A small standard deviation means the data points are clustered close to the mean, while a large one means they are widely dispersed. It is the square root of the variance and shares the same units as the data.

Common strategies include removing rows or columns with missing values, or imputing them by filling in a substitute such as the mean, median, or mode of that feature. The best choice depends on how much data is missing and why. Careless handling of missing data can bias the model.

One-hot encoding converts a categorical variable into several binary (0/1) columns, one for each category, so models can process it numerically. It avoids implying a false numerical order that would arise from simply assigning integers like 1, 2, 3 to categories. This is important for nominal categories such as colors or cities.

The scikit-learn library provides the train_test_split function from sklearn.model_selection, which randomly divides features and labels into training and test subsets. You can control the proportion with the test_size argument and ensure reproducibility with random_state. It is a standard first step in building ML models in Python.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

An epoch is one complete pass of the learning algorithm through the entire training dataset. Models are usually trained for multiple epochs so they can gradually refine their parameters over repeated exposure to the data. Too few epochs can cause underfitting, while too many can lead to overfitting.