Research · Literature and evidence
Grey Literature: Value and Risks in Applied Research
What grey literature is, why it matters for reviews and business research, where to find it, and how to judge its credibility before relying on it.
Much useful knowledge never appears in a peer-reviewed journal. Government reports, working papers, theses, conference papers, industry studies, trade association statistics and evaluation reports all contain evidence that researchers and decision-makers need. Together these are known as grey literature: material produced outside the traditional commercial and academic publishing channels. This article explains what counts as grey literature, why it matters, where to look for it and how to assess it.
What counts as grey literature
A widely cited definition, developed through the International Conference on Grey Literature, describes grey literature as material produced at all levels of government, academia, business and industry, in print and electronic formats, that is not controlled by commercial publishers (Paez, 2017). The defining feature is not quality but the publishing route. A government statistical release may be more carefully checked than many journal articles; a company white paper may be little more than marketing.
Adams et al. (2017) suggested thinking of grey literature in shades, according to how much control the producer exercises over content and how clear the expertise behind it is. The figure below sets out this idea.
Why grey literature matters
There are three main reasons to include grey literature in a review or a piece of applied research.
It reduces publication bias. Studies with positive or statistically significant results are more likely to be published in journals than studies with null results. Franco et al. (2014) followed a set of social science experiments and found that studies with strong results were far more likely to be written up and published than those with null results, most of which never left the file drawer. Hopewell et al. (2007) found that meta-analyses of health care trials that excluded grey literature tended to show larger treatment effects than those that included it. Searching beyond journals is one of the main defences against this bias (Rothstein et al., 2005).
It is often more current. Working papers and reports appear months or years before journal versions, and some never become journal articles at all. In fast-moving fields, the most recent evidence is often grey.
It covers practice. Much evidence about how organisations, markets and public programmes actually work comes from evaluations, consultancy reports and official statistics. Adams et al. (2017) argued that management and organisation studies in particular cannot ignore this material, because practitioners produce much of the knowledge about their own field. Garousi et al. (2019) made a similar case for software engineering.
Where to look
Grey literature is scattered and poorly indexed, so finding it takes deliberate effort. Common sources include:
- Government and official websites, including statistical agencies, regulators and parliamentary libraries.
- Repositories of theses and working papers, such as national thesis collections and economics working paper archives.
- Specialist databases and the websites of think tanks, charities, trade bodies and professional associations.
- Conference proceedings, which often contain early or unpublished results.
- Targeted web searches, using search operators to limit results to particular domains or file types.
- Experts and organisations in the field, who can point to reports that no search will find.
Godin et al. (2015) showed how systematic search methods can be adapted to grey literature, combining advanced web searches, targeted website searches, customised search engines and consultation with experts, and recording each step so that the search can be reported and repeated. Mahood et al. (2014) found that grey literature searching added relevant studies to their review, but also that it was time-consuming and that many sources lacked the search features of bibliographic databases. Greenhalgh and Peacock (2005) found that, in a review of complex evidence, many of the most useful sources came from following references and personal knowledge rather than from database searches. A good search plan therefore combines several routes and records each one.
Judging credibility
Because grey literature has not usually been through independent peer review, its quality varies widely and needs to be checked. Useful questions include:
- Who produced it, and with what expertise? Is the author or organisation known and qualified in the field?
- Why was it produced? Is it a neutral report, or does the producer have a commercial or political interest in the conclusion?
- How was the evidence gathered? Are the methods, sample and sources described clearly enough to judge?
- Is it dated and complete? Undated or partial documents are hard to place and harder to rely on.
- Can it be checked? Do its claims agree with other sources, and does it give enough detail for someone else to verify them?
Reporting grey literature
Reviews should state which grey sources were searched, when and how, and how the material was screened and appraised. The Cochrane handbook recommends searching trial registers, conference abstracts and other unpublished sources as a standard part of a systematic review (Higgins et al., 2019).
Conclusion
Grey literature is not second-class evidence. It fills gaps left by journals, reduces the risk that a review reflects only the most publishable results and often contains the most current evidence on practice. It also demands careful appraisal, because no editor or reviewer has done that work in advance. Used with a clear search strategy and explicit credibility checks, it makes research more complete and more honest.
References
Adams, R. J., Smart, P., & Huff, A. S. (2017). Shades of grey: Guidelines for working with the grey literature in systematic reviews for management and organizational studies. International Journal of Management Reviews, 19(4), 432–454. https://doi.org/10.1111/ijmr.12102
Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484
Garousi, V., Felderer, M., & Mäntylä, M. V. (2019). Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and Software Technology, 106, 101–121. https://doi.org/10.1016/j.infsof.2018.09.006
Godin, K., Stapleton, J., Kirkpatrick, S. I., Hanning, R. M., & Leatherdale, S. T. (2015). Applying systematic review search methods to the grey literature: A case study examining guidelines for school-based breakfast programs in Canada. Systematic Reviews, 4, Article 138. https://doi.org/10.1186/s13643-015-0125-0
Greenhalgh, T., & Peacock, R. (2005). Effectiveness and efficiency of search methods in systematic reviews of complex evidence: Audit of primary sources. BMJ, 331(7524), 1064–1065. https://doi.org/10.1136/bmj.38636.593461.68
Higgins, J. P. T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (Eds.). (2019). Cochrane handbook for systematic reviews of interventions (2nd ed.). Wiley. https://doi.org/10.1002/9781119536604
Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007). Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database of Systematic Reviews, 2007(2), Article MR000010. https://doi.org/10.1002/14651858.MR000010.pub3
Mahood, Q., Van Eerd, D., & Irvin, E. (2014). Searching for grey literature for systematic reviews: Challenges and benefits. Research Synthesis Methods, 5(3), 221–234. https://doi.org/10.1002/jrsm.1106
Paez, A. (2017). Gray literature: An important resource in systematic reviews. Journal of Evidence-Based Medicine, 10(3), 233–240. https://doi.org/10.1111/jebm.12266
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. Wiley. https://doi.org/10.1002/0470870168