Cheat sheet · Machine Learning

Logistic regression cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

Linear regression models a continuous target as a linear combination of features (minimizing squared error); logistic regression models the probability of a binary/multiclass outcome via the sigmoid/softmax of a linear score (minimizing cross-entropy). Both are interpretable Generalized Linear Models that serve as the baseline and theoretical backbone of supervised ML.

Core Formulations

Linear regression: ŷ = wᵀx + b; assumes a linear relationship between features and a continuous target, with additive (typically Gaussian) noise.
Logistic regression: p = σ(wᵀx + b), σ(z) = 1/(1 + e^(−z)); outputs a probability in (0,1), then thresholded (default 0.5) for classification — it is a CLASSIFIER despite the name.
Both are Generalized Linear Models: linear regression uses an identity link + Gaussian likelihood; logistic uses a logit link + Bernoulli likelihood.
The decision boundary of logistic regression is linear in feature space (wᵀx + b = 0); non-linear boundaries require feature engineering or basis expansion.
Log-odds interpretation: log(p/(1−p)) = wᵀx + b, so each weight wⱼ is the change in log-odds per unit increase in feature j (others fixed); e^wⱼ is the odds ratio.
Multiclass: softmax (multinomial logistic) generalizes the sigmoid: p_k = e^(z_k) / Σⱼ e^(z_j), with one weight vector per class (up to a redundant degree of freedom).

Loss Functions & Optimization

Linear regression loss: Mean Squared Error, MSE = (1/n)Σ(yᵢ − ŷᵢ)²; convex with a closed-form solution.
Normal equation: w = (XᵀX)⁻¹Xᵀy (X augmented with a column of 1s for the intercept) — exact but ~O(n·d² + d³); unstable for large or collinear feature sets, where QR/SVD solvers or gradient descent are preferred.
Logistic regression loss: binary cross-entropy (log loss) = −(1/n)Σ[yᵢ·log pᵢ + (1−yᵢ)·log(1−pᵢ)]; convex, NO closed form — solved iteratively (gradient descent, Newton/IRLS, L-BFGS, SAGA).
Why not MSE for logistic? MSE composed with the sigmoid is non-convex and produces vanishing gradients when predictions saturate; cross-entropy is the MLE-consistent, convex choice.
Both losses are derived from Maximum Likelihood Estimation under their respective likelihood assumptions (Gaussian → MSE, Bernoulli → cross-entropy).
The gradient for both has the same elegant form: ∇w = (1/n)·Xᵀ(ŷ − y), differing only in how ŷ is computed (identity for linear, sigmoid for logistic).

Assumptions

Linear regression (for valid OLS inference): linearity, independence of errors, homoscedasticity (constant error variance), approximately normal residuals, and no severe multicollinearity.
Logistic regression: linearity in the log-odds (not in y), independent observations, little multicollinearity, and a reasonably large sample per class.
Neither requires the FEATURES themselves to be normally distributed — a common myth; only residual normality matters, and only for linear-regression inference.
Multicollinearity inflates coefficient variance (check VIF > 5–10), making weights unstable and uninterpretable even when predictions stay accurate.
Assumption violations affect inference (p-values, confidence intervals) more than point prediction; for pure prediction the assumptions are softer.
Outliers and high-leverage points disproportionately affect OLS because squared error penalizes large residuals heavily.

Regularization

L2 (Ridge): adds λ‖w‖₂² — shrinks weights toward zero, handles multicollinearity, never sets weights exactly to zero; keeps the problem strictly convex and well-conditioned.
L1 (Lasso): adds λ‖w‖₁ — induces sparsity (automatic feature selection) by driving some weights exactly to zero.
Elastic Net: λ₁‖w‖₁ + λ₂‖w‖₂² — combines L1 sparsity with L2 stability, good for correlated feature groups.
In scikit-learn LogisticRegression, regularization is ON by default with strength C = 1/λ (smaller C = stronger regularization); pass penalty=None for an unregularized MLE fit.
ALWAYS standardize/scale features before regularizing, since penalties are scale-sensitive and otherwise penalize large-scale features unfairly.
Tune λ/C via cross-validation (e.g., LogisticRegressionCV, RidgeCV); regularization trades a little bias for reduced variance. The intercept is typically not penalized.

Evaluation & Interpretation

Linear: R² (variance explained), Adjusted R² (penalizes extra features), RMSE/MAE; RMSE is in target units, MAE is more robust to outliers.
Logistic: prefer probability-aware and class-balanced metrics — ROC-AUC, PR-AUC (better under heavy imbalance), F1, precision/recall, log loss; accuracy alone is misleading on imbalanced data.
Choose the decision threshold by business cost, not always 0.5; tune it via the PR/ROC curve or expected-cost analysis.
Calibration matters: logistic outputs are usually reasonably calibrated probabilities; check with reliability curves / Brier score and recalibrate (Platt/isotonic) if needed.
Coefficient interpretation: linear — Δy per unit xⱼ (holding others fixed); logistic — e^wⱼ is the odds ratio per unit xⱼ.
Standardize features first if you want to compare coefficient magnitudes as importance; otherwise differing scales confound the comparison.

Practical Tips & Variants

Strong, fast, interpretable baselines — always fit one before reaching for trees/neural nets; it sets the bar and exposes data issues.
Handle non-linearity via polynomial/interaction features, splines, or binning — keeps the model linear-in-parameters while capturing curvature.
Encode categoricals (one-hot / target encoding); watch the dummy-variable trap (drop one level to avoid perfect collinearity with the intercept).
For massive/streaming data use SGDClassifier/SGDRegressor with mini-batch gradient descent; for class imbalance use class_weight='balanced' or resampling.
Solver choice (scikit-learn): 'liblinear' (small data, L1/L2), 'lbfgs'/'newton-cg' (L2, multinomial), 'newton-cholesky' (many samples, few features), 'saga' (large data, supports L1/elastic-net and sparse input).
Related GLMs: Poisson regression (counts), softmax (multiclass); the logistic/cross-entropy loss is also the standard output layer of many neural nets.

Common pitfalls

Believing logistic regression is a regression algorithm — it is a classifier that models class probability via the logit link.
Using MSE as the logistic loss — non-convex with the sigmoid and gives vanishing gradients; use cross-entropy.
Forgetting scikit-learn's LogisticRegression is regularized by default (C=1), so 'unregularized' results differ from textbook MLE unless penalty=None.
Not scaling features before applying L1/L2 regularization, causing the penalty to unfairly target large-magnitude features.
Assuming features must be normally distributed — only residuals matter, and only for linear-regression inference, not prediction.
Reporting accuracy on imbalanced data instead of AUC/PR-AUC/F1; a 99%-accurate model can be useless on a 1%-positive class.
Interpreting correlated/multicollinear coefficients as causal or stable feature importances.
Blindly using the 0.5 threshold regardless of class imbalance or asymmetric misclassification costs.
Inverting XᵀX directly for the normal equation on collinear/large data — numerically unstable; use SVD/QR or iterative solvers.
Data leakage: fitting the scaler/encoder on the full dataset before the train/test split (fit on train only, then transform).

Interview tips

Derive both loss functions from Maximum Likelihood Estimation (Gaussian → MSE, Bernoulli → cross-entropy).
Explain why cross-entropy is preferred over MSE for logistic regression (convexity + healthy gradients).
Interpret coefficients precisely — especially the odds-ratio / log-odds meaning of logistic weights.
Contrast L1 vs L2 regularization (sparsity vs shrinkage) and when to use each, including elastic net.
State each model's assumptions and which violations hurt inference vs prediction.
Discuss multicollinearity handling (VIF, regularization, PCA) and class imbalance (class weights, resampling, threshold tuning).
Justify metric choice for a given problem (e.g., PR-AUC for rare events, RMSE vs MAE for regression).
Note that both share the gradient form Xᵀ(ŷ − y) and articulate the GLM unifying view.
Be ready to write vectorized NumPy gradient descent from scratch on a whiteboard.
Explain the bias–variance tradeoff in terms of regularization strength λ/C.

Read the full lesson Test yourself — 34 question quiz