Skip to content

Research · Research design

From Research Protocol to Documented Study

How a research protocol becomes a documented study, what a protocol should contain, why changes and deviations must be recorded, how documentation supports reproducibility and trust, and a practical record-keeping routine for applied research.

ModQu research article · 987 words · 10 references (APA 7th edition)

Download PDF

A research protocol is the written plan for a study: its questions, design, procedures and analysis, fixed before data collection begins. A documented study is the record of what was actually done. The two are rarely identical. Recruitment is slower than expected, a question is reworded after a pilot, a data source changes format, an analysis turns out to be unsuitable. None of this is a problem in itself. The problem arises when the changes are not recorded, because readers then cannot tell what was planned from what was decided after seeing the data. This article explains what a protocol should contain, how to document a study as it unfolds, and why the link between the two is central to trustworthy research.

What a protocol contains

In clinical research, the SPIRIT statement lists the items a trial protocol should include, from objectives and eligibility criteria to outcomes, sample size, randomisation, data management and statistical methods (Chan et al., 2013). The same structure suits most applied studies. At minimum, a protocol should state:

The analysis plan is the part most often left vague, and it matters most. Wicherts et al. (2016) catalogued the many decisions researchers make in analysing data, such as which observations to exclude, how to combine measures and which covariates to include, and showed how each creates an opportunity to steer results. Specifying these choices in advance removes that opportunity.

Registering the plan

Registration places the protocol, or its key elements, in a public or time-stamped record before data are collected or analysed. Nosek et al. (2018) argue that this preregistration distinguishes confirmatory tests, planned in advance, from exploratory analyses suggested by the data, and that both are valuable as long as readers can tell them apart. Clinical trials are required to register in many jurisdictions. Kaplan and Irvin (2015) found that after large trials funded by one United States agency were required to register their primary outcomes, the proportion reporting a benefit fell sharply.

From plan to record

Six-step flow from protocol to documented study: write the protocol, register it, conduct the study with a log, record deviations with reasons, analyse as planned and label extra analyses, and report against the protocol.
Figure 1. From protocol to documented study. The change log links the plan to the report, so readers can see what was planned, what changed and why. Based on Chan et al. (2013) and Nosek et al. (2018); diagram by ModQu.

Once the study begins, documentation should capture three things.

What was done. A study log records dates, procedures, data sources, versions of questionnaires and software, and who did what. For computational work, Sandve et al. (2013) recommend recording every step that produces a result, keeping scripts rather than performing manual edits, and storing the exact versions of programs and data used.

What changed. Every departure from the protocol should be noted with its date, its reason and whether it was made before or after any outcome data were seen. A change made because a pilot showed a question was misunderstood is different from a change made because the planned analysis did not produce a significant result.

What was found. Reporting guidelines such as CONSORT ask authors to state any changes to the planned outcomes after the study began, with reasons, and to report how many participants were assessed, excluded and analysed at each stage (Schulz et al., 2010). Results should be reported for the outcomes and analyses named in the protocol, including those that did not turn out as expected, with additional analyses clearly labelled as exploratory.

Why the link matters

Comparing protocols with published reports reveals how often the two diverge. Chan et al. (2004) compared the protocols of randomised trials approved by ethics committees in Denmark with the resulting publications and found that most trials had at least one primary outcome that was changed, introduced or omitted, and that outcomes with statistically significant results were more likely to be reported fully. Goldacre et al. (2019) monitored trials published in leading medical journals and found that many reported outcomes that differed from those pre-specified, often without saying so. Simmons et al. (2011) showed how undisclosed flexibility in data collection and analysis can make almost any result appear significant.

Documentation is also what makes research reproducible: the ability of others to obtain the same results from the same data and code. Peng (2011) argued that, at a minimum, published results should be accompanied by the data and code needed to reproduce them, so that others can check and build on the work.

A routine for applied research

Commercial and organisational studies rarely face formal registration requirements, but the same discipline improves their reliability and defensibility.

Conclusion

A protocol states what a study intends to do; documentation shows what it actually did. Changes are normal, but undocumented changes leave readers unable to judge whether results reflect the evidence or the choices made after seeing it. A clear protocol, a dated record of deviations and a report that relates findings back to the plan turn a study from a set of results into evidence that others can check and trust.

References

Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA, 291(20), 2457–2465. https://doi.org/10.1001/jama.291.20.2457

Chan, A.-W., Tetzlaff, J. M., Altman, D. G., Laupacis, A., Gøtzsche, P. C., Krleža-Jerić, K., Hróbjartsson, A., Mann, H., Dickersin, K., Berlin, J. A., Doré, C. J., Parulekar, W. R., Summerskill, W. S. M., Groves, T., Schulz, K. F., Sox, H. C., Rockhold, F. W., Rennie, D., & Moher, D. (2013). SPIRIT 2013 statement: Defining standard protocol items for clinical trials. Annals of Internal Medicine, 158(3), 200–207. https://doi.org/10.7326/0003-4819-158-3-201302050-00583

Goldacre, B., Drysdale, H., Dale, A., Milosevic, I., Slade, E., Hartley, P., Marston, C., Powell-Smith, A., Heneghan, C., & Mahtani, K. R. (2019). COMPare: A prospective cohort study correcting and monitoring 58 misreported trials in real time. Trials, 20, Article 118. https://doi.org/10.1186/s13063-019-3173-2

Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of null effects of large NHLBI clinical trials has increased over time. PLOS ONE, 10(8), Article e0132382. https://doi.org/10.1371/journal.pone.0132382

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114

Peng, R. D. (2011). Reproducible research in computational science. Science, 334(6060), 1226–1227. https://doi.org/10.1126/science.1213847

Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013). Ten simple rules for reproducible computational research. PLOS Computational Biology, 9(10), Article e1003285. https://doi.org/10.1371/journal.pcbi.1003285

Schulz, K. F., Altman, D. G., & Moher, D. (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. BMJ, 340, Article c332. https://doi.org/10.1136/bmj.c332

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632

Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology, 7, Article 1832. https://doi.org/10.3389/fpsyg.2016.01832