Skip to content

Statistics · Data science and machine learning

Overfitting and the Bias-Variance Trade-off

Why models that fit their data too closely predict badly, how prediction error splits into bias, variance and noise, and the practical tools that keep business models at the right level of complexity.

ModQu research article · 1,361 words · 14 references (APA 7th edition)

Download PDF

Every predictive model is a compromise. A model that is too simple misses real patterns in the data; a model that is too flexible learns patterns that are not there. The second failure, overfitting, is the more dangerous of the two because it hides behind excellent results on the data used to build the model. The bias-variance trade-off explains why it happens and points to the remedies (Geman et al., 1992; Hastie et al., 2009).

What overfitting looks like

Imagine fitting a model of purchase intention to 300 survey respondents using 60 candidate predictors, including dozens of interaction terms. With enough flexibility the model can trace almost every respondent's answer, and its R² on the sample may look impressive. But much of what it has learned is noise: the particular quirks of these 300 people that will not recur in the next sample. When the model is applied to new customers, its predictions are worse than those of a far simpler model.

Hawkins (2004) defines overfitting as using a model or procedure that is more complex than necessary, so that it fits the sample but generalises poorly. Babyak (2004) shows how common it is in regression-type models in the behavioural sciences, where many predictors, small samples and data-driven variable selection combine to produce findings that fail to replicate.

The bias-variance decomposition

To see why, consider the expected error of a model's prediction at a new point x, averaged over all the training samples we might have drawn. For squared error it splits into three parts (Geman et al., 1992):

E[(y − f̂(x))2] = Bias[f̂(x)]2 + Var[f̂(x)] + σ2

As complexity increases, bias usually falls and variance rises. Expected test error is their sum plus the noise, so it typically falls at first and then rises again. The best model sits near the bottom of that U, not at the point of best fit to the training data.

A worked simulation

Figure 1 shows a hypothetical example. A curved relationship was simulated, 25 observations were drawn with noise variance 0.12, and polynomial models of degree 1 to 14 were fitted. The process was repeated 3,000 times so that bias and variance could be measured directly.

Line chart of mean squared error against polynomial degree from 1 to 14. Squared bias falls from 0.18 at degree 1 to almost zero by degree 6. Variance rises steadily from 0.01 to 0.10. Expected test error is U-shaped, falling from 0.31 at degree 1 to a minimum of 0.153 at degree 6 and rising to 0.22 at degree 14. Training error falls continuously, from 0.29 to 0.05, and drops below the noise floor of 0.12 from degree 4 onwards.
Figure 1. Squared bias, variance, training error and expected test error for polynomial models of increasing degree, from 3,000 simulated samples of 25 observations. Illustrative data; chart by ModQu.

The straight line (degree 1) underfits: its squared bias is 0.18 and its expected test error 0.31. By degree 6 the bias has all but vanished and test error reaches its minimum of 0.15. Beyond that point every extra term adds variance without removing any bias, and by degree 14 the test error has risen to 0.22. Meanwhile, the training error keeps falling, to 0.05 at degree 14, well below the irreducible noise of 0.12. A training error below the noise level is a clear sign that the model is fitting noise.

How to detect overfitting

The decisive check is performance on data the model has not seen: a held-out test set, or cross-validation, in which the data are split into folds and each fold is predicted by a model trained on the rest (Hastie et al., 2009). A large gap between training and validation performance signals high variance; poor performance on both signals high bias. Other warning signs include coefficients that change sharply when a few cases are dropped, and implausibly large effects.

How to control complexity

Use fewer, better predictors

The simplest remedy is to match the number of parameters to the information in the data. For binary outcomes, a simulation study by Peduzzi et al. (1996) found that logistic regression estimates became unreliable with fewer than about 10 events per predictor variable. Harrell (2015) recommends deciding on candidate predictors using subject knowledge before looking at the outcome, and combining or dropping variables rather than letting automated stepwise selection choose them.

Regularise

Regularisation deliberately adds a little bias in exchange for a large reduction in variance. Ridge regression shrinks all coefficients towards zero by penalising their squared size (Hoerl & Kennard, 1970). The lasso penalises their absolute size, which shrinks some coefficients exactly to zero and so performs variable selection at the same time (Tibshirani, 1996). The strength of the penalty is a tuning parameter, normally chosen by cross-validation (James et al., 2021).

Penalise complexity when comparing models

Information criteria trade fit against the number of parameters. The Akaike information criterion (AIC) aims at good predictive accuracy (Akaike, 1974), while the Bayesian information criterion (BIC) applies a stiffer penalty that grows with sample size and tends to select smaller models (Schwarz, 1978).

Average many models

Bagging fits the same flexible model to many bootstrap samples and averages the predictions. Because the individual models' errors partly cancel, the average has much lower variance than any single model (Breiman, 1996). Random forests and many other ensemble methods build on the same idea.

Collect more data

Variance shrinks as the sample grows, so a larger sample supports a more complex model. Domingos (2012) notes that, in practice, a simple method with more data often beats a clever method with less. More data does not help with bias, however: if the model's form is wrong, its error will not fall below the bias however large the sample.

Beyond the classic U-curve

Very large modern models, such as deep neural networks with more parameters than observations, sometimes predict well despite fitting their training data perfectly. Belkin et al. (2019) describe a "double descent" pattern in which test error rises as a model approaches the point where it can just interpolate the data, then falls again as the model becomes even larger. This does not repeal the trade-off; it shows that complexity is not simply the number of parameters. For the moderate-sized data sets typical of survey and business research, the classic U-curve remains the right guide.

Practical guidance for survey and business data

Common mistakes

Key takeaways

Prediction error has three sources: bias from a model that is too rigid, variance from a model that is too flexible, and noise that no model can remove. Training error measures only how well the model memorises its sample, so judge models on new data. Keep complexity in proportion to the information available, use regularisation, information criteria or averaging to tame variance, and accept a small amount of bias when it buys a large gain in stability.

References

Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/TAC.1974.1100705

Babyak, M. A. (2004). What you see may not be what you get: A brief, nontechnical introduction to overfitting in regression-type models. Psychosomatic Medicine, 66(3), 411–421. https://doi.org/10.1097/01.psy.0000127692.23278.a9

Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116(32), 15849–15854. https://doi.org/10.1073/pnas.1903070116

Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140. https://doi.org/10.1007/BF00058655

Domingos, P. (2012). A few useful things to know about machine learning. Communications of the ACM, 55(10), 78–87. https://doi.org/10.1145/2347736.2347755

Geman, S., Bienenstock, E., & Doursat, R. (1992). Neural networks and the bias/variance dilemma. Neural Computation, 4(1), 1–58. https://doi.org/10.1162/neco.1992.4.1.1

Harrell, F. E., Jr. (2015). Regression modeling strategies: With applications to linear models, logistic and ordinal regression, and survival analysis (2nd ed.). Springer. https://doi.org/10.1007/978-3-319-19425-7

Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer. https://doi.org/10.1007/978-0-387-84858-7

Hawkins, D. M. (2004). The problem of overfitting. Journal of Chemical Information and Computer Sciences, 44(1), 1–12. https://doi.org/10.1021/ci0342472

Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1), 55–67. https://doi.org/10.1080/00401706.1970.10488634

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An introduction to statistical learning: With applications in R (2nd ed.). Springer. https://doi.org/10.1007/978-1-0716-1418-1

Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996). A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology, 49(12), 1373–1379. https://doi.org/10.1016/S0895-4356(96)00236-3

Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464. https://doi.org/10.1214/aos/1176344136

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1), 267–288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x