Skip to content

Research · Research integrity

Pre-Registration and Registered Reports

What pre-registration and Registered Reports are, why they were introduced, what the evidence shows about their effects, and how the same principles apply to commercial research.

ModQu research article · 2,083 words · 20 references (APA 7th edition)

Download PDF

A research finding is only as trustworthy as the process that produced it. Over the past two decades, researchers in psychology, medicine and the social sciences have discovered that a worrying share of published findings cannot be reproduced (Munafò et al., 2017). Ioannidis (2005) had argued that, with small samples, small effects and flexible analysis, most published findings in a field can be false. One of the main responses has been to change when researchers commit to their plans. Pre-registration means recording a study's hypotheses, methods and analysis plan in a time-stamped public registry before the data are collected or examined. Registered Reports go further: journals review and accept the study plan before the results exist. This article explains both practices, the problems they address, and what the evidence says about whether they work.

The problem: flexibility after the fact

Most research involves many decisions that are made along the way: which participants to exclude, which outcomes to emphasise, which control variables to include, and when to stop collecting data. Each choice may be reasonable on its own. Together, they give researchers a great deal of room to find a result that looks convincing.

Simmons et al. (2011) demonstrated how serious this can be. By combining a few common and individually defensible practices, such as testing several outcome measures and adding participants until a result became significant, they showed that the chance of finding a "statistically significant" effect where none existed could rise far above the nominal 5%. In one deliberately absurd demonstration, they used these practices to produce a significant result suggesting that listening to a particular song made people younger. Such practices are common: in a survey of over 2,000 psychologists, John et al. (2012) found that many admitted to at least one, such as not reporting all of a study's measures. Gelman and Loken (2014) add that the problem arises even without deliberate searching, because choices made after seeing the data depend on the data.

A related practice is HARKing, or hypothesising after the results are known: presenting a hypothesis that was formed after seeing the data as if it had been predicted in advance (Kerr, 1998). HARKing makes exploratory findings look like confirmed predictions and conceals how many other possibilities were examined.

These problems are compounded by publication bias: journals have traditionally preferred positive, novel results, so studies that find nothing are less likely to be published, the "file drawer problem" (Rosenthal, 1979). Franco et al. (2014) followed a set of social science experiments and found that those with strong results were far more likely to be written up and published than those with null results. Fanelli (2010) found that papers reporting support for their hypotheses were most common in the social sciences. The result is a literature in which effects appear larger and more reliable than they are. When the Open Science Collaboration (2015) attempted to replicate 100 studies published in leading psychology journals, far fewer of the replications produced statistically significant results than the originals had, and the replication effects were on average about half the size.

Pre-registration

Nosek et al. (2018) describe pre-registration as a way of keeping a clear line between two different kinds of research. Confirmatory research tests predictions that were made before the data were seen. Exploratory research looks for patterns in data and generates new hypotheses. Both are valuable, but they support different kinds of conclusions. Statistical tests assume that the analysis was specified in advance; when the same data are used both to generate and to test a hypothesis, the test is no longer a fair one. Pre-registration makes the distinction visible (see also Wagenmakers et al., 2012).

A pre-registration typically records:

Pre-registration has its longest history in medicine. In 2004 the International Committee of Medical Journal Editors announced that its member journals would publish clinical trials only if they had been registered in a public registry before the first patient was enrolled (De Angelis et al., 2004). The aim was to make unpublished trials and changed outcomes visible: Chan et al. (2004) had shown that outcomes reported in published trials often differed from those named in the trial protocols.

The evidence from medicine is striking. Kaplan and Irvin (2015) examined large clinical trials funded by the U.S. National Heart, Lung, and Blood Institute. Of 30 trials published before 2000, when prospective registration on ClinicalTrials.gov became expected, 17 (57%) reported a significant benefit on their primary outcome. Of 25 trials published after 2000, only 2 (8%) did. The authors concluded that the requirement to declare primary outcomes in advance was strongly associated with the decline in positive results.

Registered Reports

Pre-registration records a plan, but it does not change the incentives of journal publishing: a pre-registered study with null results may still struggle to find a home. Registered Reports address this directly. The format was launched at the journal Cortex in 2013 (Chambers, 2013) and works in two stages.

Flow diagram of a Registered Report: Stage 1 peer review of introduction, methods and analysis plan, then in-principle acceptance, then data collection, then Stage 2 review of results and conclusions, then publication regardless of outcome.
Figure 1. The two-stage Registered Report process. The journal's decision to publish is made before the results are known. Based on Chambers (2013) and Chambers and Tzavella (2022); diagram by ModQu.

At Stage 1, authors submit the introduction, methods and analysis plan. Reviewers judge whether the question is important and whether the methods are sound enough to answer it. If they are, the journal grants in-principle acceptance, committing to publish the final paper regardless of whether the hypotheses are supported, provided the authors follow the approved plan. At Stage 2, after data collection and analysis, reviewers check that the protocol was followed and that the conclusions are justified by the evidence. Additional, unplanned analyses are allowed but must be clearly labelled as exploratory.

By shifting the publication decision to before the results are known, Registered Reports remove the incentive to produce positive results and the editorial bias against null ones. Chambers and Tzavella (2022) review the format's growth from one journal to adoption across hundreds of journals in many disciplines.

What the evidence shows

Fewer positive results. Scheel et al. (2021) compared a random sample of standard psychology papers with published Registered Reports. In the standard literature, 96% of the first hypotheses tested were supported. In Registered Reports, only 44% were. The difference suggests that the standard literature substantially overstates how often hypotheses are confirmed.

Grouped bar chart. Psychology: 96% positive results in standard reports versus 44% in Registered Reports. Medicine: 57% of large NHLBI trials showed benefit before 2000 versus 8% after 2000.
Figure 2. Share of studies reporting positive results with and without advance commitment to hypotheses and outcomes. Sources: Scheel et al. (2021); Kaplan and Irvin (2015). Chart by ModQu.

Comparable or higher quality. Soderberg et al. (2021) asked researchers to compare Registered Reports with similar standard articles without knowing which was which. Registered Reports were rated as equal or better on a range of quality measures, including rigour of methods and analysis and overall quality of the paper, while being similar in novelty and creativity. This counters the concern that the format leads to dull or unimportant research.

Imperfect adherence. Pre-registration only helps if the plan is followed or deviations are disclosed. Claesen et al. (2021) examined the first generation of pre-registered studies in a leading psychology journal and found that deviations from the pre-registered plans were common, and that they were often not reported in the published paper. Pre-registration therefore works best when reviewers and readers actually compare the paper with the plan.

Writing a useful pre-registration

A pre-registration is only as useful as it is specific. A plan that says "we will compare the groups using appropriate statistical tests" leaves almost as much flexibility as no plan at all. The aim is to write the plan so precisely that someone else, given the data, could carry out exactly the same analysis and reach the same conclusion. Several practices help.

State hypotheses as testable predictions. "Customers who receive the new onboarding emails will be more satisfied" is a start; "mean satisfaction on the five-point scale at 30 days will be higher in the treatment group than the control group" is better, because it names the measure, the time point and the direction of the effect.

Fix the sample size and stopping rule in advance. Collecting data until a result becomes significant is one of the practices that Simmons et al. (2011) showed can inflate false positives. A pre-registration should state the planned sample size, how it was chosen, and what will happen if it cannot be reached.

Define the variables exactly. If an outcome is a scale built from several questions, specify which questions, how they will be combined and what happens if some are missing. If participants will be excluded, for example for failing attention checks or completing a survey implausibly fast, define the rule before seeing who would be excluded.

Name the primary outcome. Many studies measure several outcomes. Naming one or a small number as primary, and treating the rest as secondary, prevents the most favourable outcome from being chosen after the event. This was the central purpose of clinical trial registration (De Angelis et al., 2004).

Plan for the unexpected. Good pre-registrations say what will be done if data turn out not to meet the assumptions of the planned test, for instance by naming an alternative method in advance. Where some decisions genuinely cannot be made beforehand, saying so openly is better than leaving them unmentioned.

Make the plan accessible. A pre-registration needs a time-stamped, unalterable record. Public research registries provide this; clinical trials are registered in dedicated trial registries. The plan may be kept private until the study is complete, as long as the time stamp shows it was written beforehand.

The difference between pre-registration and Registered Reports

The two practices are often confused, but they address different parts of the problem. Pre-registration constrains the researcher: it makes analytic flexibility visible and separates confirmatory from exploratory work. Registered Reports also constrain the publication system: they ensure that the decision to publish does not depend on the results. A pre-registered study can still go unpublished if its results are null; a Registered Report that follows its approved protocol cannot be rejected for that reason. Pre-registration is therefore open to any researcher at any time, while Registered Reports require a journal that offers the format. Many researchers use pre-registration routinely and Registered Reports for studies where the risk of a null result is high or where an unbiased answer matters most.

Common objections

"It stifles exploration." Pre-registration does not prohibit exploratory analysis; it requires that exploratory analysis be labelled as such. Unexpected findings can still be reported, and are often the most interesting part of a study, but they are presented as hypotheses for future testing rather than as confirmed results (Nosek et al., 2018).

"Plans always change." Real research rarely goes exactly to plan. Participants drop out, a measure fails, or a better analysis becomes apparent. Pre-registration does not forbid changes; it asks that they be reported with reasons, so that readers can judge their impact.

"It doesn't suit qualitative or secondary research." Formats have been adapted for many kinds of study, including analyses of existing datasets, though the benefits are clearest for confirmatory, hypothesis-testing work.

Lessons for commercial and survey research

The principles behind pre-registration apply directly outside academia. Clients who commission survey research, A/B tests or market experiments face the same risks: many possible subgroups to examine, many possible outcome measures, and a natural pressure to find something worth reporting. A few practices borrowed from pre-registration help:

These steps cost little and make findings more credible to the people who will act on them. They also protect researchers from the most common way of fooling oneself: finding a pattern after the fact and believing it had been expected all along.

References

Chambers, C. D. (2013). Registered Reports: A new publishing initiative at Cortex. Cortex, 49(3), 609–610. https://doi.org/10.1016/j.cortex.2012.12.016

Chambers, C. D., & Tzavella, L. (2022). The past, present and future of Registered Reports. Nature Human Behaviour, 6(1), 29–42. https://doi.org/10.1038/s41562-021-01193-7

Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA, 291(20), 2457–2465. https://doi.org/10.1001/jama.291.20.2457

Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(10), Article 211037. https://doi.org/10.1098/rsos.211037

De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., Schroeder, T. V., Sox, H. C., & Van Der Weyden, M. B. (2004). Clinical trial registration: A statement from the International Committee of Medical Journal Editors. New England Journal of Medicine, 351(12), 1250–1251. https://doi.org/10.1056/NEJMe048225

Fanelli, D. (2010). "Positive" results increase down the hierarchy of the sciences. PLoS ONE, 5(4), Article e10068. https://doi.org/10.1371/journal.pone.0010068

Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484

Gelman, A., & Loken, E. (2014). The statistical crisis in science. American Scientist, 102(6), 460–465. https://doi.org/10.1511/2014.111.460

Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), Article e124. https://doi.org/10.1371/journal.pmed.0020124

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953

Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of null effects of large NHLBI clinical trials has increased over time. PLOS ONE, 10(8), Article e0132382. https://doi.org/10.1371/journal.pone.0132382

Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. https://doi.org/10.1207/s15327957pspr0203_4

Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, Article 0021. https://doi.org/10.1038/s41562-016-0021

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638

Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021). An excess of positive results: Comparing the standard psychology literature with Registered Reports. Advances in Methods and Practices in Psychological Science, 4(2). https://doi.org/10.1177/25152459211007467

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632

Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021). Initial evidence of research quality of registered reports compared with the standard publishing model. Nature Human Behaviour, 5(8), 990–997. https://doi.org/10.1038/s41562-021-01142-4

Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7(6), 632–638. https://doi.org/10.1177/1745691612463078