Logistic regression

Logistic regression is primarily used for classification problems, where the goal is to predict a discrete category (such as yes/no or 0/1) rather than a continuous number. Despite having 'regression' in its name, it estimates the probability that an input belongs to a particular class.

Logistic regression uses the sigmoid (logistic) function, defined as sigma(z) = 1 / (1 + e^(-z)). It squashes any real-valued input z into the range (0, 1), which can be interpreted as a probability.

import numpy as np
def sigmoid(z):
    return 1 / (1 + np.exp(-z))

When z = 0, the sigmoid function outputs 0.5, because sigma(0) = 1 / (1 + e^0) = 1 / (1 + 1) = 1/2. This is the point where the model is equally uncertain between the two classes.

The default threshold is 0.5: if the predicted probability is greater than or equal to 0.5, the input is classified as the positive class (1), otherwise as the negative class (0). This threshold can be adjusted depending on whether precision or recall is more important.

Logistic regression uses the log loss, also called binary cross-entropy or logistic loss. It penalizes confident wrong predictions heavily and is convex, allowing gradient descent to find the global minimum.

When MSE is combined with the sigmoid function, the resulting loss surface is non-convex, meaning gradient descent can get stuck in local minima. Log loss instead produces a convex surface, guaranteeing convergence to the global optimum.

The decision boundary is the surface that separates the predicted classes; for standard logistic regression it occurs where the predicted probability equals 0.5, i.e. where z = w·x + b = 0. In two dimensions with linear features this boundary is a straight line.

With the raw input features, the decision boundary of logistic regression is linear, because it is defined by the equation w·x + b = 0. Non-linear boundaries can be obtained by adding polynomial or other transformed features.

The log-odds, or logit, is the natural logarithm of the odds of the positive class: log(p / (1-p)). In logistic regression this quantity equals the linear combination w·x + b, which is why the model is linear in the log-odds.

The odds are the ratio of the probability of an event happening to it not happening: odds = p / (1 - p). For example, a probability of 0.8 gives odds of 0.8/0.2 = 4, meaning the event is four times as likely to occur as not.

A positive coefficient means that as the feature value increases, the log-odds (and hence the probability) of the positive class increases, holding other features constant. The larger the coefficient, the stronger the feature's positive influence on the prediction.

Gradient descent (or its variants) is commonly used to minimize the log loss by iteratively adjusting the weights in the direction that reduces the loss. Because the loss is convex, advanced solvers like L-BFGS or Newton's method are also used and converge reliably.

The LogisticRegression class from sklearn.linear_model is used. You create an instance, call .fit(X, y) to train it, and .predict(X) or .predict_proba(X) to make predictions.

from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
model.fit(X_train, y_train)
preds = model.predict(X_test)

predict_proba returns the estimated probabilities for each class, with one column per class and each row summing to 1. For binary classification it returns the probability of class 0 and class 1, letting you apply a custom threshold.

probs = model.predict_proba(X_test)
# probs[:, 1] gives probability of the positive class

Regularization adds a penalty on the size of the coefficients to the loss function, discouraging overly large weights and helping prevent overfitting. Common types are L1 (Lasso) and L2 (Ridge) penalties.

In scikit-learn, C is the inverse of regularization strength, so a smaller C means stronger regularization. Stronger regularization shrinks the coefficients more, reducing variance but potentially increasing bias.

model = LogisticRegression(C=0.01)  # strong regularization

It can be extended using multinomial (softmax) logistic regression, which generalizes the sigmoid to the softmax function across all classes, or by using a one-vs-rest (OvR) strategy that trains one binary classifier per class. Both produce probabilities over multiple categories.

The softmax function generalizes the sigmoid to multiple classes. It converts a vector of scores into a probability distribution where each value is between 0 and 1 and all values sum to 1.

import numpy as np
def softmax(z):
    e = np.exp(z - np.max(z))
    return e / e.sum()

Feature scaling (such as standardization) helps gradient-based optimizers converge faster and more reliably by putting all features on a comparable scale. It also makes regularization treat features more fairly, since the penalty depends on coefficient magnitudes.

from sklearn.preprocessing import StandardScaler
X_scaled = StandardScaler().fit_transform(X)

The bias term shifts the decision boundary and represents the log-odds of the positive class when all input features are zero. It allows the model to fit data that is not centered at the origin.

Since 0.92 is greater than 0.5, the predicted label is the positive class (1). The model is fairly confident, as the probability is well above the threshold.

The log-odds can take any real value from negative infinity to positive infinity. The sigmoid function then maps this unbounded range into the bounded probability interval (0, 1).

On imbalanced data, metrics like precision, recall, F1-score, or AUC-ROC are more informative than accuracy. Accuracy can be misleadingly high simply by predicting the majority class for every example.

MLE is the principle of choosing the model parameters that make the observed training labels most probable under the model. Minimizing the log loss is mathematically equivalent to maximizing this likelihood.

Linear regression outputs a continuous unbounded value, while logistic regression outputs a probability between 0 and 1 (via the sigmoid) used for classification. Logistic regression also uses log loss instead of squared error.

As z grows large and positive, the sigmoid output approaches (but never exactly reaches) 1. This reflects a very high confidence that the example belongs to the positive class.

Log loss includes a -log(predicted probability) term for the true class, and the logarithm tends toward infinity as the predicted probability for the correct class approaches 0. So predicting near 0 for an example that is actually positive produces a very large penalty.

The odds get multiplied by e^b for each one-unit increase in the feature, holding others constant. This is why exponentiating logistic regression coefficients gives interpretable 'odds ratios'.

A confusion matrix summarizes prediction results by showing the counts of true positives, true negatives, false positives, and false negatives. It is the basis for computing metrics like precision, recall, and accuracy.

from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, preds)

This pattern indicates overfitting: the model has learned the training data too closely, including noise, and fails to generalize to new data. Adding regularization or simplifying the model can help.

It is linear because the decision boundary is defined by a linear combination of the features (w·x + b = 0); the sigmoid only transforms this linear score into a probability and does not bend the boundary. The non-linearity applies to the output, not to the boundary in feature space.

The ROC curve plots the true positive rate against the false positive rate across all possible classification thresholds. It shows the trade-off between sensitivity and specificity, and the area under it (AUC) summarizes overall ranking performance.

Lowering the threshold makes the model classify more examples as positive, which generally increases recall (catching more true positives) but tends to decrease precision (more false positives). This trade-off lets you tune the model for the cost of each error type.

L1 regularization tends to drive some coefficients exactly to zero, effectively performing automatic feature selection. This produces a sparse model that relies only on the most useful features.

model = LogisticRegression(penalty='l1', solver='liblinear')