Cheat sheet · Deep Learning

Bias-Variance & Regularization cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

A dense reference on the bias-variance tradeoff and regularization: the formal decomposition of generalization error, how model capacity drives under/overfitting, the main regularizers (L1/L2/elastic net, dropout, early stopping, weight decay, data augmentation, normalization), and how to diagnose and fix each failure mode in practice — including modern double-descent caveats.

The Bias-Variance Decomposition

For squared loss, the expected test error at a point decomposes into three additive terms: Bias^2 + Variance + Irreducible noise. The expectation is over random draws of the training set.
Bias: error from wrong assumptions / too-simple a model (systematic). High bias = underfitting: poor on BOTH train and test, small train-test gap.
Variance: sensitivity of the learned function to the particular training sample. High variance = overfitting: low train error but a large train-test gap.
Irreducible error (sigma^2): noise inherent in the data/labels; no model can beat it. It sets the floor on achievable error.
The clean additive split is exact for MSE. For 0-1 loss and cross-entropy it is only approximate (bias and variance interact non-additively), so treat the framework as conceptual there.
Total error is minimized at an intermediate capacity — the classic U-shaped test-error curve as a function of model complexity.

Complexity, the Tradeoff, and Double Descent

Increasing capacity (more features, deeper trees, higher-degree polynomials, larger nets) lowers bias but raises variance — the core tradeoff.
Levers that move you along the curve: model family/size, polynomial degree, tree depth / min-samples-leaf, k in kNN (small k = high variance), and regularization strength.
Knobs that cut variance with little bias cost: more training data, bagging/ensembles (averaging cuts variance), feature selection, and regularization.
Boosting primarily reduces bias (sequentially fits residuals) but can overfit; bagging / Random Forests primarily reduce variance.
Modern caveat — double descent: for heavily overparameterized models (deep/wide nets), test error can fall AGAIN past the interpolation threshold, so the textbook U-curve is incomplete for large neural nets.
Practical reading: diagnose via learning curves and the train/validation gap rather than assuming where you sit on the curve.

L2 (Ridge), L1 (Lasso), and Elastic Net

Regularization adds a penalty on weight magnitude, shrinking parameters and trading a little bias for a larger reduction in variance.
L2 (Ridge): penalizes sum of squared weights; shrinks all weights smoothly toward 0 but rarely exactly to 0. Has a closed form and handles correlated features by sharing weight among them.
L1 (Lasso): penalizes sum of absolute weights; induces sparsity (drives weights exactly to 0) -> built-in feature selection. Non-differentiable at 0, solved via coordinate descent / subgradients.
Elastic Net: convex mix of L1+L2; keeps Lasso sparsity while handling groups of correlated features (Lasso alone picks one arbitrarily).
Lambda (alpha) sets strength: lambda->0 recovers the unregularized fit (high variance); lambda->inf forces weights to 0 (high bias). Tune by cross-validation.
ALWAYS standardize/scale features before L1/L2 — the penalties are scale-sensitive. By convention the intercept/bias term is NOT penalized.
Bayesian view: L2 = Gaussian (MAP) prior on weights, L1 = Laplace prior.

Regularization Beyond Penalties (esp. Deep Learning)

Early stopping: halt when validation loss stops improving; limits effective capacity and, for gradient descent on linear/quadratic losses, is provably similar to L2.
Dropout: randomly zero a fraction p of activations each step (train only); approximates averaging an ensemble of subnetworks, reducing co-adaptation. With inverted dropout (the default), survivors are scaled by 1/(1-p) DURING TRAINING, so inference needs no change and just uses the full network.
Weight decay: the optimizer-side form of L2. With Adam prefer AdamW (decoupled weight decay), since naive L2 added to the loss interacts badly with adaptive per-parameter moments.
Data augmentation: label-preserving input transforms (flips, crops, mixup, cutout, paraphrasing) increase effective data and lower variance — often the strongest regularizer in vision/NLP.
Batch/Layer normalization: stabilizes training and adds mild regularization (noise from batch statistics); LayerNorm is standard in transformers.
Other levers: label smoothing, stochastic depth, max-norm constraints, parameter sharing, and simply collecting more data.

Diagnosing and Fixing in Practice

Underfitting (high bias): high train AND validation error, small gap. Fix: add capacity/features, reduce regularization, train longer, richer model family, better feature engineering.
Overfitting (high variance): low train error, large gap to validation. Fix: more data/augmentation, increase regularization, simplify the model, dropout/early stopping, ensembling.
Learning curves (error vs training-set size): both curves converging high = bias problem (more data won't help); a persistent gap that narrows with data = variance problem (more data helps).
Validation protocol: k-fold CV for small data; a fixed held-out validation set (plus a separate test set) for large data. Never tune on the test set.
Prevent leakage: fit scalers/encoders/imputers/feature-selectors INSIDE each CV fold (use a Pipeline), or you optimistically bias the estimate.
For time series use forward-chaining / time-based splits, not random k-fold, to avoid look-ahead leakage.

Common pitfalls

Treating bias and variance as independently tunable when in practice one knob (capacity/lambda) moves both — manage the tradeoff, don't try to 'eliminate' variance.
Forgetting to scale features before L1/L2; unstandardized features make the penalty arbitrary and distort which coefficients shrink.
Penalizing the intercept/bias term (it should not be regularized), or leaving dropout/BN in training mode at inference (call model.eval()).
Assuming more data always helps — it cures high variance but does little for a high-bias (underfitting) model.
Using naive L2 with Adam instead of AdamW; the adaptive update couples the penalty incorrectly.
Data leakage: fitting scalers, imputers, or feature selectors on the full dataset before CV, inflating reported performance.
Random k-fold on time-series or grouped data, leaking future/related samples into training.
Believing the U-shaped curve always holds — overparameterized deep nets show double descent and can generalize past the interpolation point.
Mistaking Lasso's arbitrary pick among correlated features for true importance; use Elastic Net or stability selection.
Equating regularization solely with L1/L2 penalties and ignoring early stopping, augmentation, dropout, normalization, and more data.

Interview tips

Write the MSE decomposition from memory and explain why each term behaves as it does, including the irreducible noise floor.
Explain crisply WHY L1 yields sparsity and L2 does not (geometry: L1's diamond corners hit the axes; or the soft-thresholding vs proportional-shrinkage view).
State the Bayesian interpretation: L2 = Gaussian prior, L1 = Laplace prior — interviewers love this connection.
Given symptoms (train vs val error), instantly classify bias vs variance and list concrete fixes — a very common live diagnostic question.
Describe how to choose lambda (cross-validation) and how lambda maps onto the bias-variance position.
Know how regularization shows up in deep learning (dropout, weight decay/AdamW, early stopping, augmentation, BN/LN), not just in linear models.
Mention double descent to show awareness of the modern overparameterized regime; it signals you're current.
Be ready to explain why bagging reduces variance while boosting reduces bias, and how ensembles fit the tradeoff.
Explain leakage-safe validation (pipelines, per-fold preprocessing, time-based splits) — interviewers probe whether your eval is trustworthy.

Read the full lesson