Skip to content

Research · Literature and evidence

Grey Literature: Value and Risks in Applied Research

What grey literature is, why it matters for reviews and business research, where to find it, and how to judge its credibility before relying on it.

ModQu research article · 909 words · 10 references (APA 7th edition)

Download PDF

Much useful knowledge never appears in a peer-reviewed journal. Government reports, working papers, theses, conference papers, industry studies, trade association statistics and evaluation reports all contain evidence that researchers and decision-makers need. Together these are known as grey literature: material produced outside the traditional commercial and academic publishing channels. This article explains what counts as grey literature, why it matters, where to look for it and how to assess it.

What counts as grey literature

A widely cited definition, developed through the International Conference on Grey Literature, describes grey literature as material produced at all levels of government, academia, business and industry, in print and electronic formats, that is not controlled by commercial publishers (Paez, 2017). The defining feature is not quality but the publishing route. A government statistical release may be more carefully checked than many journal articles; a company white paper may be little more than marketing.

Adams et al. (2017) suggested thinking of grey literature in shades, according to how much control the producer exercises over content and how clear the expertise behind it is. The figure below sets out this idea.

Continuum of literature from white literature with high control over content, through first-tier grey literature such as government and think-tank reports, second-tier grey literature such as company reports and news articles, to third-tier grey literature such as blogs and social media posts with low control.
Figure 1. Shades of grey literature, from sources with strong control over content to sources with little. Based on Adams et al. (2017) and Garousi et al. (2019); diagram by ModQu.

Why grey literature matters

There are three main reasons to include grey literature in a review or a piece of applied research.

It reduces publication bias. Studies with positive or statistically significant results are more likely to be published in journals than studies with null results. Franco et al. (2014) followed a set of social science experiments and found that studies with strong results were far more likely to be written up and published than those with null results, most of which never left the file drawer. Hopewell et al. (2007) found that meta-analyses of health care trials that excluded grey literature tended to show larger treatment effects than those that included it. Searching beyond journals is one of the main defences against this bias (Rothstein et al., 2005).

It is often more current. Working papers and reports appear months or years before journal versions, and some never become journal articles at all. In fast-moving fields, the most recent evidence is often grey.

It covers practice. Much evidence about how organisations, markets and public programmes actually work comes from evaluations, consultancy reports and official statistics. Adams et al. (2017) argued that management and organisation studies in particular cannot ignore this material, because practitioners produce much of the knowledge about their own field. Garousi et al. (2019) made a similar case for software engineering.

Where to look

Grey literature is scattered and poorly indexed, so finding it takes deliberate effort. Common sources include:

Godin et al. (2015) showed how systematic search methods can be adapted to grey literature, combining advanced web searches, targeted website searches, customised search engines and consultation with experts, and recording each step so that the search can be reported and repeated. Mahood et al. (2014) found that grey literature searching added relevant studies to their review, but also that it was time-consuming and that many sources lacked the search features of bibliographic databases. Greenhalgh and Peacock (2005) found that, in a review of complex evidence, many of the most useful sources came from following references and personal knowledge rather than from database searches. A good search plan therefore combines several routes and records each one.

Judging credibility

Because grey literature has not usually been through independent peer review, its quality varies widely and needs to be checked. Useful questions include:

Reporting grey literature

Reviews should state which grey sources were searched, when and how, and how the material was screened and appraised. The Cochrane handbook recommends searching trial registers, conference abstracts and other unpublished sources as a standard part of a systematic review (Higgins et al., 2019).

Conclusion

Grey literature is not second-class evidence. It fills gaps left by journals, reduces the risk that a review reflects only the most publishable results and often contains the most current evidence on practice. It also demands careful appraisal, because no editor or reviewer has done that work in advance. Used with a clear search strategy and explicit credibility checks, it makes research more complete and more honest.

References

Adams, R. J., Smart, P., & Huff, A. S. (2017). Shades of grey: Guidelines for working with the grey literature in systematic reviews for management and organizational studies. International Journal of Management Reviews, 19(4), 432–454. https://doi.org/10.1111/ijmr.12102

Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484

Garousi, V., Felderer, M., & Mäntylä, M. V. (2019). Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and Software Technology, 106, 101–121. https://doi.org/10.1016/j.infsof.2018.09.006

Godin, K., Stapleton, J., Kirkpatrick, S. I., Hanning, R. M., & Leatherdale, S. T. (2015). Applying systematic review search methods to the grey literature: A case study examining guidelines for school-based breakfast programs in Canada. Systematic Reviews, 4, Article 138. https://doi.org/10.1186/s13643-015-0125-0

Greenhalgh, T., & Peacock, R. (2005). Effectiveness and efficiency of search methods in systematic reviews of complex evidence: Audit of primary sources. BMJ, 331(7524), 1064–1065. https://doi.org/10.1136/bmj.38636.593461.68

Higgins, J. P. T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (Eds.). (2019). Cochrane handbook for systematic reviews of interventions (2nd ed.). Wiley. https://doi.org/10.1002/9781119536604

Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007). Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database of Systematic Reviews, 2007(2), Article MR000010. https://doi.org/10.1002/14651858.MR000010.pub3

Mahood, Q., Van Eerd, D., & Irvin, E. (2014). Searching for grey literature for systematic reviews: Challenges and benefits. Research Synthesis Methods, 5(3), 221–234. https://doi.org/10.1002/jrsm.1106

Paez, A. (2017). Gray literature: An important resource in systematic reviews. Journal of Evidence-Based Medicine, 10(3), 233–240. https://doi.org/10.1111/jebm.12266

Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. Wiley. https://doi.org/10.1002/0470870168