Research · Literature and evidence
Systematic Reviews: Protocols, Searches and Synthesis
What a systematic review is and how one is carried out, covering protocols and registration, comprehensive searching, screening, risk-of-bias assessment, synthesis with and without meta-analysis, rating certainty and reporting.
A single study rarely settles a question. Results vary with samples, settings and methods, and the published record is affected by which studies get written up. A systematic review brings together all the relevant evidence on a focused question using explicit, reproducible methods designed to minimise bias. Petticrew and Roberts (2006) described systematic reviews as a way of making sense of large bodies of information, and noted that they are as useful for social and business questions as for medicine. This article explains the main stages: the protocol, the search and the synthesis.
Why systematic reviews matter
Narrative reviews depend on the reviewer's choice of studies, which can be selective. Systematic reviews reduce this risk by specifying in advance how studies will be found, chosen, assessed and combined. Chalmers et al. (2002) traced the history of research synthesis and showed how the failure to combine evidence systematically had led to delays in adopting effective treatments and continued use of harmful ones.
The volume of research makes synthesis essential. Bastian et al. (2010) estimated that around seventy-five trials and eleven systematic reviews were being published every day in medicine alone, far more than any practitioner could read. Systematic reviews allow decision-makers to rely on a summary of all the evidence rather than on a few memorable studies.
The stages
A systematic review moves through a fixed sequence of stages, each documented so that others can see exactly what was done. The diagram below summarises them.
The protocol
A systematic review begins with a protocol: a written plan setting out the question, eligibility criteria, search strategy, methods for screening and data extraction, approach to assessing bias and plan for synthesis. Writing the protocol before starting reduces the risk that decisions are influenced by the results. Gough et al. (2017) emphasised that the question shapes everything else, and that it should be developed with the people who will use the review.
Protocols can be registered publicly. Booth et al. (2012) described PROSPERO, an international register of systematic review protocols, which makes it possible to see whether a review is already under way and to compare a published review with its original plan.
Searching
Systematic reviews aim to find all relevant studies, not a convenient selection. Searches typically cover several databases, using carefully constructed search strings, and are supplemented by searching reference lists, citation tracking, trial registers, grey literature and contact with experts. The Cochrane Handbook recommends involving an information specialist and documenting every search in enough detail to be repeated (Higgins et al., 2019).
Screening and data extraction
Records are screened against the eligibility criteria in two stages: first titles and abstracts, then full texts. To reduce error, two reviewers usually screen independently and resolve disagreements by discussion. The flow of records through the review, from those identified to those included, is reported in a flow diagram.
For each included study, reviewers extract data using a standard form: details of participants, interventions or exposures, comparisons, outcomes, methods and results.
Assessing risk of bias
Not all studies are equally trustworthy. Reviewers assess each study's risk of bias, considering features such as how participants were allocated, whether outcome assessors knew who received the treatment, how much data were missing and whether all planned outcomes were reported. These judgements affect how much weight the review gives to each study.
Synthesis
Synthesis combines the findings. Where studies are similar enough, meta-analysis combines their results statistically. Often, studies differ too much in design, measures or populations for this. Popay et al. (2006) set out guidance for narrative synthesis, a structured approach that organises findings, explores relationships between studies and assesses the robustness of conclusions. Campbell et al. (2020) developed the SWiM reporting guideline for synthesis without meta-analysis, which asks reviewers to explain how studies were grouped, which standardised metric was used and how results were combined and presented.
Certainty of evidence
The final step is to judge how confident one can be in the findings. The GRADE approach rates the certainty of evidence for each outcome as high, moderate, low or very low, considering risk of bias, inconsistency between studies, indirectness, imprecision and publication bias (Guyatt et al., 2008). A review might find that an intervention appears effective but that the certainty is low, which is important for decision-makers to know.
Reporting
The PRISMA 2020 statement sets out what a systematic review should report, including a flow diagram and a checklist of items (Page et al., 2021). Transparent reporting allows readers to judge the review and to repeat or update it.
Quality matters
Systematic reviews vary in quality. Ioannidis (2016) argued that many published systematic reviews and meta-analyses are redundant, poorly conducted or affected by conflicts of interest. A review labelled systematic is only as good as its methods, and readers should check the protocol, search, risk-of-bias assessment and synthesis rather than relying on the label.
Keeping reviews up to date
A systematic review is a snapshot of the evidence at the date of its search. In active fields it can go out of date within a few years, as new studies are published. Bastian et al. (2010) argued that the pace of publication makes updating a constant challenge. Many organisations now plan updates at regular intervals, and some maintain living reviews, which are updated whenever important new evidence appears. Readers should always check the search date of a review before relying on it.
An applied example
A hypothetical example: a retail chain is deciding whether to extend a staff training programme and asks whether such programmes reduce staff turnover. A systematic review protocol defines the question, includes controlled studies of training in retail and hospitality, and is registered before searching begins. Searches of four databases and reference lists find 1,200 records, of which 18 studies meet the criteria. Most have moderate or high risk of bias, and their measures of turnover differ too much for meta-analysis, so the reviewers use a structured narrative synthesis. The review concludes that training is associated with lower turnover in most studies, but rates the certainty as low. The chain decides to extend the programme in a way that allows it to be evaluated properly. The figures are illustrative, but the example shows how a review informs a decision while being honest about the strength of the evidence.
Uses beyond medicine
Tranfield et al. (2003) adapted systematic review methods for management research, arguing that they could make management knowledge more reliable and useful to practitioners, while noting that management questions are often broader and evidence more varied than in medicine. A business might commission a systematic review of evidence on the effects of flexible working on productivity, for example, to inform a policy decision.
Conclusion
A systematic review answers a focused question by finding, appraising and synthesising all relevant evidence using explicit methods. Its strength lies in its protocol, comprehensive search, careful assessment of bias, appropriate synthesis and transparent reporting. Done well, it provides the most reliable basis for decisions that the available evidence can support.
References
Bastian, H., Glasziou, P., & Chalmers, I. (2010). Seventy-five trials and eleven systematic reviews a day: How will we ever keep up? PLOS Medicine, 7(9), Article e1000326. https://doi.org/10.1371/journal.pmed.1000326
Booth, A., Clarke, M., Dooley, G., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2012). The nuts and bolts of PROSPERO: An international prospective register of systematic reviews. Systematic Reviews, 1, Article 2. https://doi.org/10.1186/2046-4053-1-2
Campbell, M., McKenzie, J. E., Sowden, A., Katikireddi, S. V., Brennan, S. E., Ellis, S., Hartmann-Boyce, J., Ryan, R., Shepperd, S., Thomas, J., Welch, V., & Thomson, H. (2020). Synthesis without meta-analysis (SWiM) in systematic reviews: Reporting guideline. BMJ, 368, Article l6890. https://doi.org/10.1136/bmj.l6890
Chalmers, I., Hedges, L. V., & Cooper, H. (2002). A brief history of research synthesis. Evaluation & the Health Professions, 25(1), 12–37. https://doi.org/10.1177/0163278702025001003
Gough, D., Oliver, S., & Thomas, J. (Eds.). (2017). An introduction to systematic reviews (2nd ed.). SAGE.
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008). GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ, 336(7650), 924–926. https://doi.org/10.1136/bmj.39489.470347.AD
Higgins, J. P. T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (Eds.). (2019). Cochrane handbook for systematic reviews of interventions (2nd ed.). Wiley. https://doi.org/10.1002/9781119536604
Ioannidis, J. P. A. (2016). The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. The Milbank Quarterly, 94(3), 485–514. https://doi.org/10.1111/1468-0009.12210
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, Article n71. https://doi.org/10.1136/bmj.n71
Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Blackwell. https://doi.org/10.1002/9780470754887
Popay, J., Roberts, H., Sowden, A., Petticrew, M., Arai, L., Rodgers, M., Britten, N., Roen, K., & Duffy, S. (2006). Guidance on the conduct of narrative synthesis in systematic reviews: A product from the ESRC Methods Programme. Lancaster University.
Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14(3), 207–222. https://doi.org/10.1111/1467-8551.00375