Predicting numbers (regression)

Regression is a type of machine learning where a computer learns to predict a continuous number from labeled example data. For example, if we show a computer the sizes of houses and their selling prices, it can learn to predict the price of a new house it has never seen. The output is always a number — like temperature, price, or height — rather than a category.

A line of best fit is a straight line drawn through a scatter plot of data points that best represents the overall trend in the data. It tries to be as close as possible to all the data points at once — not perfectly touching any single one. In linear regression, the computer finds this exact line automatically using mathematics.

Features are the input values used to make a prediction — for example, a student's number of study hours. Labels are the output values we want to predict — for example, that student's exam score. The regression model learns the relationship between features and labels from many training examples.

Training a regression model means showing the computer many paired examples of inputs and correct outputs so it can learn the pattern connecting them. The computer adjusts its internal numbers — called parameters — until its predictions become as accurate as possible on the training data. After training, the model can predict outputs for new inputs it has never seen before.

Predicting tomorrow's temperature in degrees Celsius is a regression problem because the answer is a continuous number rather than a category. Other regression examples include predicting a car's fuel efficiency in km/l or estimating a person's height from their age. Problems where the answer is a category — like 'spam' or 'not spam' — are called classification, not regression.

The y-intercept 'b' is the predicted value of the output y when the input feature x equals zero. For example, if a model predicts monthly electricity bill from kilowatt-hours used and b = 50, then even with zero measured usage the predicted bill starts at ₹50 — perhaps representing a fixed monthly connection charge. It is where the regression line crosses the vertical y-axis.

A positive correlation means that as one variable increases, the other tends to increase as well. For example, taller people tend to weigh more — height and weight have a positive correlation. Regression models use these correlations between features and labels to learn their prediction patterns, though correlation alone does not prove that one thing causes the other.

A negative slope means that as the input feature increases in value, the model predicts that the output will decrease. For example, a model predicting exam score from hours spent watching TV instead of studying would have a negative slope: more TV time predicts a lower score. The steeper (more negative) the slope, the stronger the inverse relationship between input and output.

The slope 'm' tells us how much the predicted output (y) changes when the input feature (x) increases by exactly 1 unit. For example, if m = 3 in a model predicting test score from study hours, each extra hour of study is predicted to add 3 marks. A positive slope means the label increases with the feature, while a negative slope means it decreases.

A residual is the difference between the actual correct value and a model's predicted value for a specific data point (actual − predicted). If the model predicts 80 but the real answer is 75, the residual is 75 − 80 = −5. The negative sign tells us the model overestimated. Analysing residuals helps us understand where the model makes mistakes and whether there are systematic patterns in the errors that could be improved.

Mean Absolute Error is calculated by finding the difference between each prediction and the actual value, taking the absolute value (ignoring positive or negative sign), and averaging all those differences. It tells us, on average, how far off the model's predictions are from the true values in the original units. For example, an MAE of 5 on a model predicting exam scores means predictions are off by about 5 marks on average.

# Calculate MAE manually in Python
errors = [abs(pred - actual) for pred, actual in zip(predictions, actuals)]
mae = sum(errors) / len(errors)
print(f'MAE: {mae}')

We split data so the model is trained on one portion but evaluated on a completely separate portion it has never seen before. This tells us whether the model has truly learned the underlying pattern or simply memorised the training examples. If we tested on the same data we trained on, we would get an overly optimistic score that would not reflect how the model performs in the real world.

Overfitting happens when a model learns the training data so perfectly — including its random noise and quirks — that it performs poorly on any new data it has not seen before. Think of it like a student who memorises exact past exam answers but cannot solve any question worded even slightly differently. An overfit model has very low training error but much higher error on the test set.

Underfitting occurs when a model is too simple to capture the true pattern in the data, resulting in poor predictions on both the training and test sets. For example, fitting a straight line through data that actually follows a curve is a classic case of underfitting. The solution is usually to use a more complex model type or to add more informative features.

Interpolation means making a prediction within the range of values the model was trained on, which is generally reliable. Extrapolation means predicting far outside that range, which is much riskier because the underlying pattern may change beyond the training data boundaries. For example, a model trained on exam scores for 10–20 hours of study may give very unreliable predictions for 50 hours of study.

Each relevant feature gives the model more useful information to base its predictions on, reducing uncertainty and helping it capture more of the real pattern. For example, predicting a house price using only its size is less accurate than using size, number of bedrooms, and location together. However, adding irrelevant features can hurt the model by introducing noise, so only features that logically connect to the output should be included.

Mean Squared Error squares each residual before averaging them, which means large errors are penalised much more heavily than small ones. For example, an error of 10 contributes 100 to the MSE, while an error of 2 contributes only 4 — making MSE very sensitive to outliers. MAE, by contrast, simply averages the absolute errors without squaring, treating all error magnitudes more proportionally.

# MSE penalises big errors more than MAE
mse = sum((p - a)**2 for p, a in zip(preds, actuals)) / len(actuals)
mae = sum(abs(p - a) for p, a in zip(preds, actuals)) / len(actuals)

Feature scaling puts all input features on a similar numerical range so that no single feature unfairly dominates the model just because its numbers happen to be larger. For example, if one feature is height in centimetres (150–200) and another is income in rupees (10,000–100,000), without scaling the model would pay far more attention to income. Models that use gradient descent converge much faster and more reliably when features are scaled to comparable ranges.

Outliers are data points that lie far away from the general pattern, and they can pull the regression line toward them, reducing accuracy for the majority of the data. For example, if one house in a dataset sold for ten times its true value due to a data entry error, the model could learn incorrect patterns from that single point. Carefully identifying and handling outliers — by fixing errors or removing them — is an important step in preparing data for regression.

Regression predicts a continuous number — like exact temperature, price, or height — while classification predicts which category something belongs to, like 'cat' or 'dog', or 'pass' or 'fail'. For example, predicting a student's exact exam score is regression, but predicting whether they will pass or fail is classification. Both are forms of supervised learning where the model learns from labeled training examples.

If the training data reflects historical biases — for example, lower loan approval rates for certain groups due to past discriminatory practices — the regression model will learn and reproduce those biases in future predictions. This is called algorithmic bias, and it can cause unfair outcomes in high-stakes areas like credit scoring, hiring, or healthcare. It is important to audit model predictions across different demographic groups and question whether the training data itself is fair and representative.

You import `LinearRegression` from `sklearn.linear_model`, create an instance, call `.fit(X_train, y_train)` to train on labeled data, and then call `.predict(X_test)` to generate predictions for new inputs. Scikit-learn handles all the mathematical optimisation automatically, so you can build and use regression models without coding gradient descent from scratch. This makes it a very popular starting point for learning and applying regression in Python.

from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(X_train, y_train)          # train the model
predictions = model.predict(X_test)  # make predictions

Gradient descent is the method a computer uses to find the best parameters (slope and intercept) for a regression model by making many small, guided adjustments. It starts with random parameter values, measures how wrong the predictions are using a loss function, then nudges the parameters slightly in the direction that reduces the error. This process repeats thousands of times until the error is as small as possible — like a ball rolling downhill to find the lowest point in a valley.

A loss function is a mathematical formula that measures how wrong a model's predictions are on the training data, producing a single summary number called the loss. During training, the model uses an optimisation method like gradient descent to make this number as small as possible. Mean Squared Error (MSE) is a common loss function for regression — it squares each residual before averaging, which penalises large errors more heavily than small ones.

R-squared measures what proportion of the variation in the label is explained by the model's input features, expressed as a value between 0 and 1. An R² of 0.85 means the model explains 85% of the variation in the output, with the remaining 15% left unexplained by the features used. A higher R² generally indicates a better-fitting model, though it should always be used alongside other metrics for a complete picture.

Bias refers to errors caused by a model being too simple — it misses important patterns and underfits. Variance refers to errors caused by a model being too sensitive to the specific training data — it captures noise as well as real patterns and overfits. The tradeoff is that reducing one type of error often increases the other, so the goal is to find the right model complexity that balances both for the best performance on genuinely new data.

Polynomial regression fits a curved line through data by using squared, cubed, or higher powers of the input feature alongside the original feature. It is more useful than linear regression when the true relationship between input and output is curved rather than straight. For example, the relationship between a car's speed and its fuel consumption forms a U-shaped curve, which a straight line would fail to capture accurately.

Cross-validation divides the dataset into several equal groups called folds, then trains and evaluates the model multiple times — each time using a different fold as the test set and the remaining folds as training data. The final score is the average across all these runs, giving a much more reliable estimate of real-world performance than any single split. It is especially valuable for small datasets where a single random split might be unrepresentatively lucky or unlucky.

Data leakage happens when information from the test set — or from future data the model should not know about — accidentally influences the training process, making the model appear far more accurate than it really is. For example, scaling features using statistics calculated across the whole dataset including the test set is a subtle but common form of leakage. A model trained with leaked data gives unrealistically optimistic evaluation results but performs poorly when deployed in the real world.

If we repeatedly check the test set to tune settings like model complexity or learning rate, we are effectively giving the model indirect information about the test data — the model ends up optimised for that specific test set rather than for truly unseen data. This is called test set contamination, and it causes evaluation scores to be misleadingly high. Instead, a separate validation set or cross-validation should be used for all tuning, keeping the test set completely untouched until the very final evaluation.

Regularisation is a technique that adds a penalty to the loss function for having very large parameter values, discouraging the model from becoming overly complex and overfitting the training data. Ridge regression (L2) adds a penalty proportional to the sum of squared parameters, while Lasso regression (L1) penalises the sum of absolute parameter values. The result is a simpler, more generalised model that performs better on new unseen data.

Multiple linear regression extends simple linear regression by using two or more input features simultaneously instead of just one. The model learns a separate weight for each feature and combines them to produce a single numerical prediction. For example, predicting a house price might use size, number of bedrooms, and age of the house — each contributing its own weighted amount to the final price estimate.

# price = w1*size + w2*bedrooms + w3*age + b
from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X_train, y_train)  # X_train has 3 columns (features)

Yes, a neural network can solve regression problems by using an output neuron with no activation function (or a linear activation), which allows it to output any continuous number. The network is trained to minimise a regression loss function like Mean Squared Error, exactly like linear regression. The key advantage is that neural networks can automatically learn very complex non-linear relationships from data that a simple linear regression line could never capture.

Regression gives unreliable predictions when the true relationship between features and label is not the shape the model assumes — for example, using a straight line for curved data. Predictions also become unreliable when important explanatory features are missing, when the training data contains many uncorrected outliers, or when the model is asked to predict far outside the range of values it was trained on (extrapolation). Recognising these limitations is essential for knowing when to trust a model's output.