Skip to content

Research · Communicating research

Research Limitations: How to Report Them Without Undermining Findings

Why limitations are under-reported, what a useful limitations section contains, and how to describe weaknesses in a study so that readers can judge how far its findings can be trusted.

ModQu research article · 1,178 words · 11 references (APA 7th edition)

Download PDF

Every study has limitations. Samples are smaller or narrower than ideal, measures are imperfect, some participants drop out, and the setting may differ from the places where the findings will be applied. Reporting these weaknesses is part of reporting the study accurately. Yet many writers treat the limitations section as a formality or fear that it will weaken their conclusions. This article reviews the evidence on how limitations are reported and sets out a practical approach that informs readers without discounting sound findings.

How often limitations are acknowledged

Ioannidis (2007) examined the first 50 research articles published in 2005 in each of six highly cited research journals and two newer open-access journals, 400 articles in all. Only 17% used at least one word denoting limitations, only 1% mentioned a limitation in the abstract, and very few had a separate limitations section. Of the author instructions and editorial policies of the 25 most-cited journals, only one encouraged authors to discuss limitations; most emphasised importance and novelty instead.

A later survey found more acknowledgement but similar gaps. ter Riet et al. (2013) examined 300 biomedical articles published in 2007 in 30 journals. In 27% of them, no limitations were mentioned at all. The remaining articles acknowledged a median of three limitations, but only 5% mentioned a limitation in the abstract. Articles reporting industry-supported studies used fewer expressions of uncertainty than other articles.

Bar chart from two surveys. Ioannidis 2007, 400 articles: 17% used any word denoting limitations, 1% mentioned limitations in the abstract. ter Riet and colleagues 2013, 300 articles: 73% acknowledged at least one limitation, 5% mentioned one in the abstract.
Figure 1. How often published articles acknowledged limitations, in the full text and in the abstract. The two surveys used different samples and definitions and are not directly comparable. Sources: Ioannidis (2007); ter Riet et al. (2013). Chart by ModQu.

The abstract matters because it is the part of an article that most readers see. A limitation mentioned only in the discussion will be missed by anyone who reads only the summary.

Why limitations are under-reported

The reasons are mostly about incentives. Journals and funders reward novelty and confident conclusions (Ioannidis, 2007). Writers may worry that reviewers will use an acknowledged weakness as a reason to reject a paper, or that readers will dismiss the findings. Commercial researchers face pressure to give clients a clear answer.

These concerns are understandable but largely misplaced. Experiments by van der Bles et al. (2020) found that stating the uncertainty around an estimate as a numerical range reduced trust in the figure and in its source only slightly; larger decreases came mainly from vague verbal statements of uncertainty.

What a useful limitations section contains

Puhan et al. (2012) proposed a set of principles for discussing limitations in biomedical reports, which apply equally well in other fields:

Reporting guidelines take the same view. The CONSORT statement for randomised trials asks authors to discuss "trial limitations, addressing sources of potential bias, imprecision, and, if relevant, multiplicity of analyses" (Schulz et al., 2010), and the STROBE statement for observational studies asks for a discussion of limitations that takes account of potential bias or imprecision, including its direction and magnitude (von Elm et al., 2007).

Describe consequences, not just weaknesses

The most common failing of limitations sections is listing weaknesses without saying what they mean. "The sample was not random" leaves the reader to work out the consequences. A more useful version explains them:

Respondents were recruited from the company's newsletter subscribers, who are likely to be more engaged than customers in general. Satisfaction scores are therefore probably higher than they would be for all customers. The comparison between the two service options is less likely to be affected, because both groups were recruited in the same way.

This version names the limitation, states the likely direction of bias, and distinguishes the findings it affects from those it does not. It weakens one conclusion and leaves another standing, which is exactly the information a reader needs.

Four-step diagram for writing a limitation statement: name the limitation, state the likely direction and size of bias, say which findings it affects and which it does not, and say what was done about it or what would resolve it.
Figure 2. Four parts of a useful limitation statement. Based on the principles in Puhan et al. (2012); diagram by ModQu.

Keep limitations in proportion

A long list of minor or generic limitations can be as unhelpful as none. Statements such as "more research is needed" or "the results may not generalise" apply to almost every study and tell the reader nothing. Writers should concentrate on the limitations that could change a reader's interpretation, and say which they consider most serious.

Limitations should also be judged against the claims actually made. A study of one hospital does not need to apologise for not representing all hospitals if it does not claim to. Being precise about the scope of a conclusion in the first place, for example "in this sample of mid-sized retailers in Ontario", reduces the need for later qualification.

The GRADE approach to rating the certainty of evidence, used widely in health research and guideline development, provides a useful model. It considers risk of bias, inconsistency between studies, indirectness of the evidence, imprecision and publication bias, and rates the overall certainty as high, moderate, low or very low (Guyatt et al., 2008). A single study can borrow this vocabulary to tell readers how much confidence each main finding deserves.

Limitations and spin

Omitting limitations is one form of spin: presenting results as more favourable than the evidence supports. Boutron et al. (2010) found that in reports of randomised trials whose primary outcomes were not statistically significant, the conclusions of more than half of the abstracts presented the results more favourably than the data justified. A systematic review by Chiu et al. (2017) found spin across many kinds of biomedical study, and Sumner et al. (2014) found that exaggeration in health news often began in university press releases. Clear limitations, stated near the conclusions they qualify, are a direct protection against this.

Where to put limitations

In academic articles, limitations usually appear in the discussion, often under their own subheading. In reports for decision-makers, the main limitations belong in the summary as well, close to the findings they affect. A practical rule is that any limitation serious enough to change a decision should appear wherever the finding it qualifies appears.

Reported this way, limitations do not undermine sound findings. They show which findings are sound, how far they reach and what would make them stronger.

References

Boutron, I., Dutton, S., Ravaud, P., & Altman, D. G. (2010). Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA, 303(20), 2058–2064. https://doi.org/10.1001/jama.2010.651

Chiu, K., Grundy, Q., & Bero, L. (2017). 'Spin' in published biomedical literature: A methodological systematic review. PLOS Biology, 15(9), Article e2002173. https://doi.org/10.1371/journal.pbio.2002173

Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008). GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ, 336(7650), 924–926. https://doi.org/10.1136/bmj.39489.470347.AD

Ioannidis, J. P. A. (2007). Limitations are not properly acknowledged in the scientific literature. Journal of Clinical Epidemiology, 60(4), 324–329. https://doi.org/10.1016/j.jclinepi.2006.09.011

Puhan, M. A., Akl, E. A., Bryant, D., Xie, F., Apolone, G., & ter Riet, G. (2012). Discussing study limitations in reports of biomedical studies: The need for more transparency. Health and Quality of Life Outcomes, 10, Article 23. https://doi.org/10.1186/1477-7525-10-23

Rothwell, P. M. (2005). External validity of randomised controlled trials: "To whom do the results of this trial apply?" The Lancet, 365(9453), 82–93. https://doi.org/10.1016/S0140-6736(04)17670-8

Schulz, K. F., Altman, D. G., & Moher, D. (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. BMJ, 340, Article c332. https://doi.org/10.1136/bmj.c332

Sumner, P., Vivian-Griffiths, S., Boivin, J., Williams, A., Venetis, C. A., Davies, A., Ogden, J., Whelan, L., Hughes, B., Dalton, B., Boy, F., & Chambers, C. D. (2014). The association between exaggeration in health related science news and academic press releases: Retrospective observational study. BMJ, 349, Article g7015. https://doi.org/10.1136/bmj.g7015

ter Riet, G., Chesley, P., Gross, A. G., Siebeling, L., Muggensturm, P., Heller, N., Umbehr, M., Vollenweider, D., Yu, T., Akl, E. A., Brewster, L., Dekkers, O. M., Mühlhauser, I., Richter, B., Singh, S., Goodman, S., & Puhan, M. A. (2013). All that glitters isn't gold: A survey on acknowledgment of limitations in biomedical studies. PLOS ONE, 8(11), Article e73623. https://doi.org/10.1371/journal.pone.0073623

van der Bles, A. M., van der Linden, S., Freeman, A. L. J., & Spiegelhalter, D. J. (2020). The effects of communicating uncertainty on public trust in facts and numbers. Proceedings of the National Academy of Sciences, 117(14), 7672–7683. https://doi.org/10.1073/pnas.1913678117

von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007). The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: Guidelines for reporting observational studies. The Lancet, 370(9596), 1453–1457. https://doi.org/10.1016/S0140-6736(07)61602-X