Skip to content

Statistics · Inference

P-Values: What They Mean and What They Do Not

What a p-value actually measures, the most common ways it is misread in business and survey reports, and how to report test results so that decision-makers are not misled.

ModQu research article · 751 words · 6 references (APA 7th edition)

Download PDF

The p-value is the most widely reported number in applied statistics and one of the most misunderstood. Concerned by its misuse, the American Statistical Association issued its first formal statement on a statistical method, setting out what p-values can and cannot tell us (Wasserstein & Lazar, 2016).

What a p-value is

A p-value is the probability, calculated assuming the null hypothesis and every other assumption of the statistical model are true, of obtaining a result at least as extreme as the one observed. In short, it measures how surprising the data would be if there were no real effect.

In a hypothetical packaging test, purchase intention is 44.5% for a new pack and 40.0% for the current one, with 1,000 respondents per group. The test statistic is z = 2.04 and the two-sided p-value is 0.042. The correct reading is: if the two packs truly performed the same, a difference of 4.5 points or more in either direction would occur in about 4% of samples this size. Small p-values signal that the data fit poorly with the null model.

What a p-value is not

Greenland et al. (2016) list more than twenty common misinterpretations, and Goodman (2008) describes a "dirty dozen". The most damaging in business reporting are these.

Why p-values behave as they do

When the null hypothesis is true, every p-value between 0 and 1 is equally likely. A p-value below 0.05 then occurs in 5% of studies purely by chance, which is exactly the Type I error rate. When there is a real effect, small p-values become more common, but how much more depends on power.

Two histograms of p-values from many repeated studies, in bins of width 0.05. In the left panel, where there is no real effect, every bin holds 5% of studies, so the bars form a flat line. In the right panel, where a real effect exists and the test has 50% power, 50% of p-values fall below 0.05, about 12% fall between 0.05 and 0.10, and the bars shrink steadily to under 1% near 1.
Figure 1. Distribution of two-sided p-values across many repeated studies with no real effect (left) and with a real effect detected at 50% power (right). Illustrative data; chart by ModQu.

Figure 1 shows that even with a genuine effect, half of the studies at 50% power would miss significance, and repeating a study can easily turn p = 0.03 into p = 0.30. A single p-value is a noisy summary of evidence.

Practical guidance for reporting

Key takeaways

A p-value tells you how compatible the data are with a specific null model, nothing more. It is not the probability that a hypothesis is true, not a measure of effect size and not proof of no effect when it is large. Read alongside effect sizes, confidence intervals and context, it is a useful check on sampling noise; read alone, it misleads.

References

Gelman, A., & Stern, H. S. (2006). The difference between "significant" and "not significant" is not itself statistically significant. The American Statistician, 60(4), 328–331. https://doi.org/10.1198/000313006X152649

Goodman, S. N. (2008). A dirty dozen: Twelve p-value misconceptions. Seminars in Hematology, 45(3), 135–140. https://doi.org/10.1053/j.seminhematol.2008.04.003

Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. https://doi.org/10.1007/s10654-016-0149-3

Sellke, T., Bayarri, M. J., & Berger, J. O. (2001). Calibration of p values for testing precise null hypotheses. The American Statistician, 55(1), 62–71. https://doi.org/10.1198/000313001300339950

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108

Wasserstein, R. L., Schirm, A. L., & Lazar, N. A. (2019). Moving to a world beyond "p < 0.05". The American Statistician, 73(sup1), 1–19. https://doi.org/10.1080/00031305.2019.1583913