Skip to content

Statistics · Tests and models

Analysis of Variance (ANOVA) and Its Extensions

How analysis of variance compares the means of three or more groups, how to read the F ratio and effect sizes, and when to use Welch, factorial, repeated-measures and covariance extensions.

ModQu research article · 1,091 words · 11 references (APA 7th edition)

Download PDF

Market researchers often need to compare more than two groups at once. Do three packaging designs differ in willingness to pay? Does satisfaction vary across five regions? Running a separate t-test for every pair of groups inflates the chance of a false positive, because each extra test is another opportunity for a Type I error. Analysis of variance (ANOVA), developed by R. A. Fisher and set out for practitioners in his 1925 textbook (Fisher, 1925), answers the overall question with a single test: is there more variation between the group means than we would expect from the variation within the groups?

The logic of the F ratio

ANOVA splits the total variation in an outcome into two parts. The between-group part reflects how far each group mean lies from the grand mean. The within-group part reflects how far individual respondents lie from their own group mean, which is treated as noise. Each part is divided by its degrees of freedom to give a mean square, and the test statistic is their ratio:

F = MSbetween ÷ MSwithin

If the population means are all equal, F should be close to 1. A large F means the group means are further apart than within-group noise alone would explain. The null hypothesis is that all population means are equal; the alternative is that at least one differs. A significant F therefore says that some difference exists, not where it lies.

A worked example

Suppose a hypothetical concept test shows each of three packaging designs to 50 respondents and asks how much they would pay. The group means are $62, $68 and $74, and the standard deviation within each group is $20. The between-group sum of squares is 50 × (6² + 0² + 6²) = 3,600 on 2 degrees of freedom, so MSbetween = 1,800. The within-group mean square is 20² = 400 on 147 degrees of freedom. The result is F(2, 147) = 1,800 ÷ 400 = 4.5, with p = .013.

Two panels of dot plots of willingness to pay for packaging designs A, B and C, each with 50 respondents and group means of 62, 68 and 74 dollars against a grand mean of 68 dollars. In panel A the within-group standard deviation is 10 dollars, the clouds of points are tight and F(2, 147) equals 18.0 with p below .001. In panel B, the worked example, the standard deviation is 20 dollars, the clouds overlap heavily and F(2, 147) equals 4.5 with p equal to .013.
Figure 1. The same three group means produce a much larger F ratio when respondents within each group are less spread out. Illustrative data; chart by ModQu.

Figure 1 shows why the within-group spread matters. With the same means but half the spread, F rises to 18.0. Statistical significance is not the same as importance, so ANOVA results should always be reported with an effect size. Eta squared, the share of total variation explained by group membership, is 3,600 ÷ 62,400 = 0.058 here. Omega squared corrects its small upward bias and gives about 0.045. Lakens (2013) and Olejnik and Algina (2003) explain which version to report for different designs.

Assumptions and how robust ANOVA is

Classical ANOVA assumes independent observations, roughly normal residuals and equal variances across groups. An extensive review of earlier empirical studies by Glass et al. (1972) found the F test fairly robust to non-normality, but sensitive to unequal variances when group sizes also differ. Independence matters most of all: repeated answers from the same respondent, or clustered samples, break the test unless the design is modelled.

Unequal variances are common in business data. Welch (1951) proposed a version of the F test that does not assume equal variances, and Delacre et al. (2019) argue that Welch's F should be the default, since it loses little power when variances are equal and protects the error rate when they are not. Tests of equal variances, such as the median-based version of Levene's test studied by Brown and Forsythe (1974), are informative but should not act as a gatekeeper for choosing the main test.

Finding which groups differ

A significant overall F is usually followed by post hoc comparisons. Tukey's (1949) honestly significant difference procedure compares every pair of means while holding the familywise error rate at the chosen level. When variances or group sizes are unequal, the Games–Howell procedure is preferred (Games & Howell, 1976). For a small number of planned comparisons decided before seeing the data, specific contrasts are more powerful than testing every pair.

Extensions of the basic model

All of these are special cases of the general linear model, which means they can also be fitted as a regression with dummy-coded groups.

Practical guidance for survey data

Weighted or clustered survey samples need design-based tests, because ordinary ANOVA assumes simple random sampling and understates standard errors when the design effect is large. Survey software tests group differences through regression with design-based Wald tests (Lumley, 2010). Plan sample sizes per group, not just in total, since the smallest group limits precision. Report group means with confidence intervals alongside the F test so that readers can see the size of differences in the original units, such as dollars or rating points.

Common mistakes

Key takeaways

ANOVA tests whether group means differ by comparing variation between groups with variation within them. Report the F ratio together with an effect size and group means with intervals. Prefer Welch's F when variances may differ, follow a significant result with Tukey or Games–Howell comparisons, and choose factorial, repeated-measures, covariance or design-based extensions to match how the data were collected.

References

Brown, M. B., & Forsythe, A. B. (1974). Robust tests for the equality of variances. Journal of the American Statistical Association, 69(346), 364–367. https://doi.org/10.1080/01621459.1974.10482955

Delacre, M., Leys, C., Mora, Y. L., & Lakens, D. (2019). Taking parametric assumptions seriously: Arguments for the use of Welch's F-test instead of the classical F-test in one-way ANOVA. International Review of Social Psychology, 32(1), Article 13. https://doi.org/10.5334/irsp.198

Fisher, R. A. (1925). Statistical methods for research workers. Oliver and Boyd.

Games, P. A., & Howell, J. F. (1976). Pairwise multiple comparison procedures with unequal n's and/or variances: A Monte Carlo study. Journal of Educational Statistics, 1(2), 113–125. https://doi.org/10.3102/10769986001002113

Glass, G. V., Peckham, P. D., & Sanders, J. R. (1972). Consequences of failure to meet assumptions underlying the fixed effects analyses of variance and covariance. Review of Educational Research, 42(3), 237–288. https://doi.org/10.3102/00346543042003237

Greenhouse, S. W., & Geisser, S. (1959). On methods in the analysis of profile data. Psychometrika, 24(2), 95–112. https://doi.org/10.1007/BF02289823

Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, Article 863. https://doi.org/10.3389/fpsyg.2013.00863

Lumley, T. (2010). Complex surveys: A guide to analysis using R. Wiley. https://doi.org/10.1002/9780470580066

Olejnik, S., & Algina, J. (2003). Generalized eta and omega squared statistics: Measures of effect size for some common research designs. Psychological Methods, 8(4), 434–447. https://doi.org/10.1037/1082-989X.8.4.434

Tukey, J. W. (1949). Comparing individual means in the analysis of variance. Biometrics, 5(2), 99–114. https://doi.org/10.2307/3001913

Welch, B. L. (1951). On the comparison of several mean values: An alternative approach. Biometrika, 38(3/4), 330–336. https://doi.org/10.2307/2332579