Research · Research design
From Research Protocol to Documented Study
How a research protocol becomes a documented study, what a protocol should contain, why changes and deviations must be recorded, how documentation supports reproducibility and trust, and a practical record-keeping routine for applied research.
A research protocol is the written plan for a study: its questions, design, procedures and analysis, fixed before data collection begins. A documented study is the record of what was actually done. The two are rarely identical. Recruitment is slower than expected, a question is reworded after a pilot, a data source changes format, an analysis turns out to be unsuitable. None of this is a problem in itself. The problem arises when the changes are not recorded, because readers then cannot tell what was planned from what was decided after seeing the data. This article explains what a protocol should contain, how to document a study as it unfolds, and why the link between the two is central to trustworthy research.
What a protocol contains
In clinical research, the SPIRIT statement lists the items a trial protocol should include, from objectives and eligibility criteria to outcomes, sample size, randomisation, data management and statistical methods (Chan et al., 2013). The same structure suits most applied studies. At minimum, a protocol should state:
- the research questions and, where relevant, the hypotheses;
- the design and the population, with eligibility and sampling methods;
- the measures, with the exact wording of key questions or the definition of key variables;
- the procedures for collecting, cleaning and storing data;
- the planned analysis, including the primary outcome, how missing data will be handled and which comparisons will be made; and
- the timeline and the people responsible for each part.
The analysis plan is the part most often left vague, and it matters most. Wicherts et al. (2016) catalogued the many decisions researchers make in analysing data, such as which observations to exclude, how to combine measures and which covariates to include, and showed how each creates an opportunity to steer results. Specifying these choices in advance removes that opportunity.
Registering the plan
Registration places the protocol, or its key elements, in a public or time-stamped record before data are collected or analysed. Nosek et al. (2018) argue that this preregistration distinguishes confirmatory tests, planned in advance, from exploratory analyses suggested by the data, and that both are valuable as long as readers can tell them apart. Clinical trials are required to register in many jurisdictions. Kaplan and Irvin (2015) found that after large trials funded by one United States agency were required to register their primary outcomes, the proportion reporting a benefit fell sharply.
From plan to record
Once the study begins, documentation should capture three things.
What was done. A study log records dates, procedures, data sources, versions of questionnaires and software, and who did what. For computational work, Sandve et al. (2013) recommend recording every step that produces a result, keeping scripts rather than performing manual edits, and storing the exact versions of programs and data used.
What changed. Every departure from the protocol should be noted with its date, its reason and whether it was made before or after any outcome data were seen. A change made because a pilot showed a question was misunderstood is different from a change made because the planned analysis did not produce a significant result.
What was found. Reporting guidelines such as CONSORT ask authors to state any changes to the planned outcomes after the study began, with reasons, and to report how many participants were assessed, excluded and analysed at each stage (Schulz et al., 2010). Results should be reported for the outcomes and analyses named in the protocol, including those that did not turn out as expected, with additional analyses clearly labelled as exploratory.
Why the link matters
Comparing protocols with published reports reveals how often the two diverge. Chan et al. (2004) compared the protocols of randomised trials approved by ethics committees in Denmark with the resulting publications and found that most trials had at least one primary outcome that was changed, introduced or omitted, and that outcomes with statistically significant results were more likely to be reported fully. Goldacre et al. (2019) monitored trials published in leading medical journals and found that many reported outcomes that differed from those pre-specified, often without saying so. Simmons et al. (2011) showed how undisclosed flexibility in data collection and analysis can make almost any result appear significant.
Documentation is also what makes research reproducible: the ability of others to obtain the same results from the same data and code. Peng (2011) argued that, at a minimum, published results should be accompanied by the data and code needed to reproduce them, so that others can check and build on the work.
A routine for applied research
Commercial and organisational studies rarely face formal registration requirements, but the same discipline improves their reliability and defensibility.
- Write a short protocol before fieldwork, agreed with the client or sponsor, naming the primary questions and the main analysis.
- Keep a dated change log in the project folder, recording each change and its reason.
- Version the instruments and the data, so that every result can be traced to the questionnaire and file that produced it.
- Script the analysis rather than editing data by hand.
- Report against the protocol, stating what was planned, what changed and which findings are exploratory.
Conclusion
A protocol states what a study intends to do; documentation shows what it actually did. Changes are normal, but undocumented changes leave readers unable to judge whether results reflect the evidence or the choices made after seeing it. A clear protocol, a dated record of deviations and a report that relates findings back to the plan turn a study from a set of results into evidence that others can check and trust.
References
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA, 291(20), 2457–2465. https://doi.org/10.1001/jama.291.20.2457
Chan, A.-W., Tetzlaff, J. M., Altman, D. G., Laupacis, A., Gøtzsche, P. C., Krleža-Jerić, K., Hróbjartsson, A., Mann, H., Dickersin, K., Berlin, J. A., Doré, C. J., Parulekar, W. R., Summerskill, W. S. M., Groves, T., Schulz, K. F., Sox, H. C., Rockhold, F. W., Rennie, D., & Moher, D. (2013). SPIRIT 2013 statement: Defining standard protocol items for clinical trials. Annals of Internal Medicine, 158(3), 200–207. https://doi.org/10.7326/0003-4819-158-3-201302050-00583
Goldacre, B., Drysdale, H., Dale, A., Milosevic, I., Slade, E., Hartley, P., Marston, C., Powell-Smith, A., Heneghan, C., & Mahtani, K. R. (2019). COMPare: A prospective cohort study correcting and monitoring 58 misreported trials in real time. Trials, 20, Article 118. https://doi.org/10.1186/s13063-019-3173-2
Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of null effects of large NHLBI clinical trials has increased over time. PLOS ONE, 10(8), Article e0132382. https://doi.org/10.1371/journal.pone.0132382
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114
Peng, R. D. (2011). Reproducible research in computational science. Science, 334(6060), 1226–1227. https://doi.org/10.1126/science.1213847
Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013). Ten simple rules for reproducible computational research. PLOS Computational Biology, 9(10), Article e1003285. https://doi.org/10.1371/journal.pcbi.1003285
Schulz, K. F., Altman, D. G., & Moher, D. (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. BMJ, 340, Article c332. https://doi.org/10.1136/bmj.c332
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology, 7, Article 1832. https://doi.org/10.3389/fpsyg.2016.01832