Linear regression

Linear regression is a supervised learning algorithm used to model the relationship between one or more input features and a continuous numeric target by fitting a straight line (or hyperplane). It predicts output values, such as house prices or temperatures, based on the learned linear relationship between inputs and output.

The simple linear regression equation is y = mx + b (often written y = b0 + b1*x), where m (or b1) is the slope, b (or b0) is the intercept, x is the input feature, and y is the predicted output. The slope measures how much y changes per unit change in x, and the intercept is the predicted value when x is zero.

The slope coefficient represents the expected change in the target variable y for a one-unit increase in the input feature x, holding other features constant. A positive slope means y increases as x increases, while a negative slope means y decreases as x increases.

The intercept is the predicted value of the target variable y when all input features are equal to zero. Geometrically, it is the point where the regression line crosses the y-axis. Sometimes it has no real-world meaning if x = 0 is outside the realistic range of data.

Linear regression most commonly minimizes the Mean Squared Error (MSE), which is the average of the squared differences between predicted and actual values. Squaring the errors penalizes larger mistakes more heavily and makes the function smooth and differentiable for optimization.

The least squares method finds the line that minimizes the sum of the squared vertical distances (residuals) between the observed data points and the predicted values on the line. By minimizing squared residuals, it produces the best-fitting line in terms of overall error.

A residual is the difference between the actual observed value and the value predicted by the regression model (residual = actual - predicted). Residuals measure how far off each prediction is, and analyzing them helps assess whether the model fits the data well.

Gradient descent is an iterative optimization algorithm that adjusts the model's parameters (slope and intercept) step by step in the direction that reduces the cost function. In linear regression it is used to find the coefficient values that minimize the Mean Squared Error, especially for large datasets where the closed-form normal equation is expensive.

for i in range(epochs):
    y_pred = m*x + b
    dm = -2 * np.mean(x * (y - y_pred))
    db = -2 * np.mean(y - y_pred)
    m -= lr * dm
    b -= lr * db

The learning rate controls the size of the steps taken toward the minimum of the cost function during each iteration. If it is too large, the algorithm may overshoot and fail to converge; if it is too small, training becomes very slow and may take many iterations to reach the minimum.

Simple linear regression uses a single input feature to predict the target, fitting a straight line. Multiple linear regression uses two or more input features and fits a hyperplane, allowing the model to capture relationships involving several variables at once.

R-squared measures the proportion of the variance in the target variable that is explained by the model. On training data it ranges from 0 to 1, where an R-squared of 0.8 means 80% of the variation in the target is explained by the model, with higher values indicating a better fit. Note that on new/unseen data R-squared can even be negative if the model predicts worse than the mean.

An R-squared of 0 means the model explains none of the variance in the target variable and performs no better than simply predicting the mean of y for every input. This indicates the features have no linear predictive power for the target as modeled.

Linear regression assumes that the relationship between the input features and the target variable is linear, meaning the target can be expressed as a weighted sum of the features plus a constant. If the true relationship is strongly nonlinear, a plain linear model will fit poorly.

Multicollinearity occurs when two or more input features are highly correlated with each other. It makes the estimated coefficients unstable and hard to interpret, because the model cannot clearly separate the individual effect of each correlated feature on the target.

Overfitting happens when a model learns the noise and random fluctuations in the training data rather than the underlying pattern, often due to too many features or excessive complexity. An overfit model performs very well on training data but poorly on new, unseen data.

Ridge regression adds a penalty equal to the sum of the squared coefficients to the cost function, shrinking the coefficients toward zero. This reduces model variance and helps prevent overfitting, especially when features are correlated, by discouraging overly large weights.

Lasso adds a penalty equal to the sum of the absolute values of the coefficients, which can shrink some coefficients exactly to zero, effectively performing feature selection. Ridge uses squared coefficients and shrinks weights toward zero but rarely makes them exactly zero, keeping all features.

Residuals can be positive or negative, so simply summing them would let positive and negative errors cancel out, hiding the true error. Squaring makes all errors positive, prevents cancellation, and penalizes larger errors more strongly, while keeping the function smooth and differentiable.

The LinearRegression class from sklearn.linear_model is used for ordinary least squares linear regression. You create an instance, call fit(X, y) to train it, and then use predict(X) to generate predictions on new data.

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X_train, y_train)
preds = model.predict(X_test)

The learned coefficients (slopes) are stored in model.coef_ and the intercept is stored in model.intercept_. The coef_ attribute is an array with one value per feature, while intercept_ is a single scalar value.

print(model.coef_)
print(model.intercept_)

Mean Absolute Error (MAE) averages the absolute differences between predicted and actual values, treating all errors proportionally. Mean Squared Error (MSE) averages the squared differences, so it penalizes large errors much more heavily and is more sensitive to outliers than MAE.

Feature scaling puts all features on a comparable range so that gradient descent converges faster and more reliably, since features with very different magnitudes can otherwise cause uneven, zig-zagging steps. It also makes regularization penalties apply fairly across all coefficients.

A curved or systematic pattern in the residual plot suggests that the linearity assumption is violated, meaning a straight line does not adequately capture the relationship. This indicates the model is underfitting and may need nonlinear terms or a different model.

The normal equation is a closed-form analytical solution that directly computes the optimal coefficients without iteration, given by theta = (X^T X)^(-1) X^T y. It is convenient for small to medium datasets but becomes computationally expensive when the number of features is very large because it requires inverting a matrix.

import numpy as np
theta = np.linalg.inv(X.T @ X) @ X.T @ y

Because least squares minimizes squared errors, outliers with large residuals are penalized heavily and can pull the regression line strongly toward them. This can distort the slope and intercept, making the line a poor fit for the majority of the data.

Homoscedasticity requires that the variance of the residuals (errors) is roughly constant across all levels of the predicted values. If the spread of residuals increases or decreases systematically (heteroscedasticity), it violates this assumption and can make coefficient estimates and confidence intervals unreliable.

In simple linear regression, the slope equals the Pearson correlation coefficient (r) multiplied by the ratio of the standard deviation of y to the standard deviation of x (slope = r * std_y / std_x). So the slope has the same sign as the correlation, and a stronger correlation generally relates to a steeper standardized relationship.

Plain R-squared never decreases when you add more features, even useless ones, which can make complex models look better than they are. Adjusted R-squared penalizes adding features that do not improve the model, so it gives a fairer comparison between models with different numbers of predictors.

High bias means the model is too simple to capture the underlying pattern in the data, leading to systematic errors and underfitting. Such a model performs poorly on both the training and test sets because it makes overly simplistic assumptions about the relationship.

Yes, linear regression can capture nonlinear relationships by adding transformed features such as polynomial terms (x^2, x^3) or interaction terms, while the model remains linear in its coefficients. This technique, called polynomial regression, fits curves to data while still using the linear regression framework.

from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)

Evaluating on the training data alone can give an overly optimistic estimate because the model has already seen that data and may have memorized it. A separate test set provides an unbiased estimate of how the model generalizes to new, unseen data, which is what we ultimately care about.

from sklearn.model_selection import train_test_split
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2)

RMSE is the square root of the MSE, which brings the error metric back into the same units as the target variable. This makes RMSE easier to interpret directly, for example as a typical prediction error measured in dollars or kilograms rather than squared units.

Convergence is indicated when the cost function stops decreasing meaningfully and stabilizes at a minimum, so further iterations produce only tiny changes in the parameters. At this point the gradients are near zero, meaning the parameters have reached values close to the optimum.

Linear regression assumes that the residuals are independent of one another, meaning the error for one observation does not predict the error for another. This assumption is commonly violated in time-series data, where consecutive errors are correlated (autocorrelation), which can lead to underestimated standard errors.