Skip to content

Surveys · Ethics and law

Disclosure Standards for Publishing Survey Results

Readers can judge a published survey result only if they know how the survey was done. This article explains the disclosure standards set by professional bodies, what information should accompany any published survey, why it matters and how to apply the standards in practice.

ModQu research article · 1,125 words · 12 references (APA 7th edition)

Download PDF

A headline that says "62 per cent of people support the new policy" tells readers very little on its own. Who was asked? How were they chosen? What exactly was the question? When was the survey done, and how many people answered? Without this information, readers cannot tell a careful survey from a self-selected online vote. Disclosure standards set out the information that should be made available whenever survey results are published, so that others can evaluate them.

Why disclosure matters

Survey results can be affected by many choices: the population and sampling method, the mode, the question wording and order, the dates of fieldwork, the weighting and the treatment of people who did not answer. Two surveys on the same topic can produce different results because of these choices, not because opinion has changed. When polls miss election results, as in the United Kingdom in 2015 and the United States in 2016, investigations depend on detailed information about methods to find the causes (Sturgis et al., 2018; Kennedy et al., 2018). The real error of published polls is also larger than their reported margins of error suggest, which readers can only judge if methods are disclosed (Shirani-Mehr et al., 2018).

Disclosure also protects the public from misleading research. A survey with a biased sample or leading questions can be used to support almost any claim. Requiring disclosure makes such problems visible.

What the standards require

Several professional bodies set disclosure standards. The AAPOR Code requires researchers to disclose specified information when they publish results or allow others to publish them (American Association for Public Opinion Research, 2021). ESOMAR and the World Association for Public Opinion Research (WAPOR) publish a joint guideline on opinion polls and published surveys (ESOMAR & WAPOR, 2014). In the United Kingdom, members of the British Polling Council commit to publishing details of their methods and computer tables of results (British Polling Council, n.d.). The international standard ISO 20252 includes requirements for reporting research (International Organization for Standardization, 2019).

Although the details differ, the standards agree on a core set of items, shown in Figure 1.

Cards listing the core items to disclose with published survey results: sponsor and conductor, population, sample method and size, mode and dates, question wording and weighting and precision.
Figure 1. Core information to disclose with published survey results. Diagram by ModQu.

Disclosure for non-probability samples

Many published surveys now use opt-in online panels rather than probability samples. Standards require researchers to say so clearly and not to report a conventional margin of error as if the sample were random. Where a measure of precision is given, its basis, such as a model-based credibility interval, should be explained. Readers should also be told how panel members were recruited and what adjustments were made.

Reporting subgroups, changes and error

Disclosure applies to how results are described as well as to methods. Results for subgroups, such as young adults or people in one region, rest on smaller samples and carry larger margins of error, so the number of respondents in each subgroup should be given. Claims that opinion has changed should be based on surveys that used the same methods, and differences should be described as real only when they exceed what sampling variation alone could produce. Reports should also avoid suggesting more accuracy than the survey can provide, for example by quoting results to a decimal place. Sampling error is only one part of total survey error, which also includes errors of coverage, nonresponse and measurement that no margin of error captures (Biemer & Lyberg, 2003).

Disclosure in practice

Most news reports cannot include every item, so standards usually require a short statement with the report and full details available elsewhere, for example on the research company's website. A typical short statement reads: "The survey was conducted online by Example Research for the Daily News among 1,512 adults aged 18 and over in Canada between 3 and 6 March. Results are weighted by age, sex, region and education. Full tables and question wording are available at example.com." Figure 2 shows how often such items are reported in news coverage of polls.

Horizontal bar chart of the share of news reports of polls that mention each disclosure item, from sample size at 81 per cent to weighting at 14 per cent.
Figure 2. Share of news reports of polls mentioning each disclosure item (illustrative values). Chart by ModQu.

Journalists play a part as well. Guidance for reporters lists questions to ask before reporting any poll, covering most of the disclosure items (Gawiser & Witt, 2007). Research on the reporting of polls has long found that news coverage often omits important methodological details (Lavrakas & Traugott, 2000).

Transparency beyond the minimum

Some organisations go further. AAPOR's Transparency Initiative recognises organisations that commit to routine disclosure of their methods. Some publish full questionnaires, computer tables and even anonymised data sets. This reflects a wider movement towards openness in research, in which methods and data are shared so that results can be checked and reproduced (Nosek et al., 2015).

Enforcement

Professional bodies can investigate complaints that members failed to disclose methods, and may publicly censure those who refuse. Their reach is limited to members, and many published surveys come from organisations outside any association. In some countries, election laws require published pre-election polls to include certain information, and some restrict publication in the final days before a vote.

An example

A charity publishes a press release stating that "eight in ten parents worry about screen time". A journalist asks for the details and learns that the survey was an online questionnaire promoted on the charity's social media pages, completed by 640 self-selected parents, with no weighting. The question asked respondents whether they agreed that "too much screen time is harming children". The journalist reports the finding with these details, noting that the sample was self-selected and the question was worded in one direction (illustrative example).

Conclusion

Disclosure standards ensure that published survey results come with the information needed to judge them: who did the survey, whom it represents, how respondents were chosen, what was asked, when, how and with what precision. Following them is a mark of professional research, and asking for them is the best protection readers have against misleading figures.

References

American Association for Public Opinion Research. (2021). AAPOR code of professional ethics and practices. AAPOR. https://aapor.org/standards-and-ethics/

American Association for Public Opinion Research. (2023). Standard definitions: Final dispositions of case codes and outcome rates for surveys (10th ed.). AAPOR. https://aapor.org/standards-and-ethics/standard-definitions/

Biemer, P. P., & Lyberg, L. E. (2003). Introduction to survey quality. Wiley. https://doi.org/10.1002/0471458740

British Polling Council. (n.d.). Officers, objects and rules. British Polling Council.

ESOMAR & WAPOR. (2014). ESOMAR/WAPOR guideline on opinion polls and published surveys. ESOMAR.

Gawiser, S. R., & Witt, G. E. (2007). 20 questions a journalist should ask about poll results (3rd ed.). National Council on Public Polls.

International Organization for Standardization. (2019). Market, opinion and social research, including insights and data analytics — Vocabulary and service requirements (ISO Standard No. 20252:2019).

Kennedy, C., Blumenthal, M., Clement, S., Clinton, J. D., Durand, C., Franklin, C., McGeeney, K., Miringoff, L., Olson, K., Rivers, D., Saad, L., Witt, G. E., & Wlezien, C. (2018). An evaluation of the 2016 election polls in the United States. Public Opinion Quarterly, 82(1), 1–33. https://doi.org/10.1093/poq/nfx047

Lavrakas, P. J., & Traugott, M. W. (Eds.). (2000). Election polls, the news media, and democracy. Chatham House.

Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., . . . Yarkoni, T. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425. https://doi.org/10.1126/science.aab2374

Shirani-Mehr, H., Rothschild, D., Goel, S., & Gelman, A. (2018). Disentangling bias and variance in election polls. Journal of the American Statistical Association, 113(522), 607–614. https://doi.org/10.1080/01621459.2018.1448823

Sturgis, P., Kuha, J., Baker, N., Callegaro, M., Fisher, S., Green, J., Jennings, W., Lauderdale, B. E., & Smith, P. (2018). An assessment of the causes of the errors in the 2015 UK general election opinion polls. Journal of the Royal Statistical Society: Series A (Statistics in Society), 181(3), 757–781. https://doi.org/10.1111/rssa.12329