Bias-variance tradeoff

Bias is the error introduced by approximating a complex real-world relationship with a model that is too simple. High bias means the model makes strong, often wrong assumptions and consistently misses the true pattern, leading to underfitting. For example, fitting a straight line to clearly curved data produces high bias.

Variance is the amount by which a model's predictions would change if it were trained on a different training dataset. High variance means the model is very sensitive to the specific training data, capturing noise as if it were signal, which causes overfitting. Such a model performs well on training data but poorly on unseen data.

High bias and underfitting describe the same problem from different angles: the model is too simple to capture the underlying structure of the data. An underfit model has high error on both the training set and the test set because it cannot represent the true relationship. Increasing model complexity is one way to reduce this bias.

High variance leads to overfitting: the model learns the noise and idiosyncrasies of the training data rather than the general pattern. An overfit model shows very low training error but high test error because it does not generalize. Reducing variance (e.g., via regularization or more data) helps fix overfitting.

The bias-variance tradeoff is the tension between two sources of error: reducing bias usually increases variance, and reducing variance usually increases bias. As model complexity grows, bias falls but variance rises, so the goal is to find a complexity level that minimizes total error on unseen data. It explains why neither the simplest nor the most complex model is usually best.

Expected test error decomposes into bias squared, variance, and irreducible error. Bias squared and variance can be reduced by choosing better models, but irreducible error (caused by inherent noise in the data) cannot be removed by any model. This decomposition guides where to focus improvement efforts.

# Conceptual decomposition
# E[(y - f_hat(x))^2] = Bias[f_hat]^2 + Var[f_hat] + sigma^2
# sigma^2 is the irreducible noise variance

Irreducible error is the part of the prediction error caused by inherent randomness or noise in the data and unmeasured factors. Because it does not come from the model, no amount of tuning, more data, or better algorithms can remove it. It sets a lower bound on the achievable test error.

As complexity increases, bias generally decreases because the model can fit more intricate patterns, while variance generally increases because the model becomes more sensitive to the training data. This opposing movement is the core of the tradeoff. The best model sits where the combined error is lowest, not at either extreme.

This is a sign of high variance (overfitting): the model fits the training data very well but fails to generalize to new data. The large gap between low training error and high test error is the key clue. Remedies include adding more data, simplifying the model, or applying regularization.

This indicates high bias (underfitting): the model is too simple, so it performs poorly even on the data it was trained on. Since both errors are high and close together, adding more data will not help much. Increasing model complexity or adding informative features is the right fix.

A higher-degree polynomial has more parameters and flexibility, so it can bend to pass through individual training points, including noise. This makes the fitted curve change drastically with different training samples, which is exactly high variance. Although bias keeps falling, the rising variance can make test error worse beyond an optimal degree.

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
# Higher degree -> lower bias, higher variance
model = make_pipeline(PolynomialFeatures(degree=15), LinearRegression())

Regularization adds a penalty on large coefficients, which shrinks the model toward simpler solutions. This reduces variance by making the model less sensitive to the training data, at the cost of a small increase in bias. The regularization strength is tuned to find the sweet spot that minimizes test error.

from sklearn.linear_model import Ridge
# Larger alpha -> more shrinkage -> lower variance, higher bias
model = Ridge(alpha=1.0)

Adding more training data generally reduces variance because the model sees a more representative sample and is less swayed by noise in any single example. It has little direct effect on bias, since bias is about the model's inherent assumptions. This is why 'get more data' is a common cure for overfitting but not for underfitting.

High bias comes from the model being too simple to represent the true relationship, not from a lack of data. Feeding more examples to an overly simple model still leaves it unable to capture the pattern, so both training and test error stay high. The fix is a more expressive model or better features, not more data.

A learning curve plots training and validation error against the amount of training data. If both curves converge at a high error, the model has high bias; if there is a large persistent gap with low training error, the model has high variance. This visual tool tells you whether to add complexity or add data/regularization.

from sklearn.model_selection import learning_curve
# Compare train vs validation scores as training size grows

Cross-validation gives a more reliable estimate of how the model generalizes to unseen data, which mainly reflects variance-related performance. By averaging error over several folds, it reduces the chance of being misled by one lucky or unlucky train/test split. It is widely used to select model complexity and hyperparameters that balance bias and variance.

from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X, y, cv=5)

A small k (like k=1) gives a very flexible model with low bias but high variance, since predictions depend heavily on a single nearby point. Increasing k averages over more neighbors, which smooths predictions, lowering variance but raising bias. Choosing k is therefore a direct bias-variance balancing act.

from sklearn.neighbors import KNeighborsClassifier
# small k: low bias/high variance; large k: high bias/low variance
model = KNeighborsClassifier(n_neighbors=15)

A shallow tree makes coarse splits, giving high bias and low variance, while a very deep tree can isolate individual training points, giving low bias but high variance. Deep trees tend to overfit the training data. Limiting depth (pruning) is a way to trade a little bias for a large reduction in variance.

from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier(max_depth=4)  # limit depth to reduce variance

Bagging trains many models on different bootstrap samples of the data and averages their predictions. Averaging cancels out much of the random fluctuation in individual high-variance models, lowering overall variance without greatly increasing bias. Random Forests add feature randomness to make the trees less correlated, boosting this effect.

from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=200)  # averaging lowers variance

Boosting builds models sequentially, with each new model focusing on the errors of the previous ones, which mainly reduces bias by gradually fitting the data more closely. However, because it keeps adding complexity, boosting can increase variance and overfit if run for too many iterations. Early stopping and a small learning rate help keep variance in check.

from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(learning_rate=0.1, n_estimators=100)

A very complex model can drive bias to near zero but usually has high variance, fitting noise and generalizing poorly to new data. Total expected error is the sum of bias squared, variance, and irreducible error, so minimizing only bias can still leave a large total error. The best model balances both terms rather than maximizing flexibility.

The decomposition comes from expanding the expected squared error (mean squared error), and the squaring of the difference between the expected prediction and the true value naturally produces a bias-squared term. Using the square keeps the contribution non-negative and in the same units as variance. This makes the three terms directly additive.

Consistent, systematic offset in the same direction indicates high bias, because bias measures the average difference between predicted and true values. Variance, by contrast, would show as scattered predictions that change with the sample. A systematic error pattern points to a model that is too simple or makes wrong assumptions.

Tightly clustered darts mean low variance because the throws are consistent, but being far from the bullseye means high bias since they systematically miss the target. This represents an underfit model that is precise but inaccurate. The ideal is tight clustering centered on the bullseye (low bias, low variance).

Widely scattered darts mean high variance because the throws are inconsistent, while being centered on the bullseye on average means low bias. This corresponds to a flexible model that is right on average but unstable for any single training set. More data or averaging (ensembling) can tighten the scatter.

As training continues, a neural network's training error keeps falling, but at some point validation error starts rising as the model begins to memorize noise (rising variance). Early stopping halts training at the point of lowest validation error, trading a little extra bias for reduced variance. It is a simple and effective form of regularization.

Adding more training data, applying regularization, or using dropout/ensembling can reduce variance while keeping the same kind of model. These approaches make the model less sensitive to the particular training set. They are preferred when the model family is appropriate but is overfitting.

Good generalization means the model performs well on unseen data, which requires keeping both bias and variance reasonably low so total expected error is small. A well-generalizing model captures the true pattern (low bias) without chasing noise (low variance). Cross-validation is used to confirm that performance holds on data the model has not seen.

Each extra feature gives the model more freedom to fit random patterns in the training data, especially when the feature carries no real signal. This added flexibility makes predictions more sensitive to the specific training sample, raising variance. Feature selection or regularization can counter this effect.

The degree-1 line has higher bias because it is too simple to capture any curvature, underfitting the data. The degree-20 curve has higher variance because it wiggles to fit noise and would change drastically with new data, overfitting. The ideal model usually lies somewhere between these extremes.

Capacity refers to the range of functions a model can represent; low-capacity models tend to have high bias, and high-capacity models tend to have high variance. Increasing capacity reduces bias but raises variance, so there is an optimal capacity that minimizes generalization error. Choosing capacity is essentially choosing where to sit on the tradeoff curve.

Total error is the sum of bias squared, variance, and irreducible error, so different combinations of bias and variance can add up to the same total. One model might be simple (high bias, low variance) while another is complex (low bias, high variance), yet both yield similar overall error. Knowing the split matters because it tells you what to fix to improve further.

Dropout randomly disables a fraction of neurons during each training step, preventing the network from relying too heavily on any single neuron or co-adapted path. This reduces variance by acting like training an ensemble of many smaller networks, improving generalization. It is a regularization technique that may slightly increase bias in exchange.

The optimal complexity is the point where the sum of bias squared and variance (and thus total expected test error) is at its minimum. To the left of it the model underfits (high bias), and to the right it overfits (high variance). Techniques like cross-validation are used to locate this sweet spot empirically.