Surveys · Ethics and law
Disclosure Standards for Publishing Survey Results
Readers can judge a published survey result only if they know how the survey was done. This article explains the disclosure standards set by professional bodies, what information should accompany any published survey, why it matters and how to apply the standards in practice.
A headline that says "62 per cent of people support the new policy" tells readers very little on its own. Who was asked? How were they chosen? What exactly was the question? When was the survey done, and how many people answered? Without this information, readers cannot tell a careful survey from a self-selected online vote. Disclosure standards set out the information that should be made available whenever survey results are published, so that others can evaluate them.
Why disclosure matters
Survey results can be affected by many choices: the population and sampling method, the mode, the question wording and order, the dates of fieldwork, the weighting and the treatment of people who did not answer. Two surveys on the same topic can produce different results because of these choices, not because opinion has changed. When polls miss election results, as in the United Kingdom in 2015 and the United States in 2016, investigations depend on detailed information about methods to find the causes (Sturgis et al., 2018; Kennedy et al., 2018). The real error of published polls is also larger than their reported margins of error suggest, which readers can only judge if methods are disclosed (Shirani-Mehr et al., 2018).
Disclosure also protects the public from misleading research. A survey with a biased sample or leading questions can be used to support almost any claim. Requiring disclosure makes such problems visible.
What the standards require
Several professional bodies set disclosure standards. The AAPOR Code requires researchers to disclose specified information when they publish results or allow others to publish them (American Association for Public Opinion Research, 2021). ESOMAR and the World Association for Public Opinion Research (WAPOR) publish a joint guideline on opinion polls and published surveys (ESOMAR & WAPOR, 2014). In the United Kingdom, members of the British Polling Council commit to publishing details of their methods and computer tables of results (British Polling Council, n.d.). The international standard ISO 20252 includes requirements for reporting research (International Organization for Standardization, 2019).
Although the details differ, the standards agree on a core set of items, shown in Figure 1.
- Sponsor and conductor: who paid for the survey and who carried it out.
- Population: the group the survey is meant to represent, such as adults aged 18 and over in Canada.
- Sample: how respondents were selected, including whether the sample was a probability sample or drawn from an opt-in panel, and the number of respondents.
- Mode and dates: how the survey was carried out, such as by telephone or online, and when.
- Questions: the exact wording of the questions reported, and of any preceding questions that might have affected answers.
- Weighting: the variables used to weight the results.
- Precision: a measure of sampling error for probability samples, or an explanation of why one is not given for non-probability samples.
- Response: response rates or participation rates where they can be calculated, using standard definitions (American Association for Public Opinion Research, 2023).
Disclosure for non-probability samples
Many published surveys now use opt-in online panels rather than probability samples. Standards require researchers to say so clearly and not to report a conventional margin of error as if the sample were random. Where a measure of precision is given, its basis, such as a model-based credibility interval, should be explained. Readers should also be told how panel members were recruited and what adjustments were made.
Reporting subgroups, changes and error
Disclosure applies to how results are described as well as to methods. Results for subgroups, such as young adults or people in one region, rest on smaller samples and carry larger margins of error, so the number of respondents in each subgroup should be given. Claims that opinion has changed should be based on surveys that used the same methods, and differences should be described as real only when they exceed what sampling variation alone could produce. Reports should also avoid suggesting more accuracy than the survey can provide, for example by quoting results to a decimal place. Sampling error is only one part of total survey error, which also includes errors of coverage, nonresponse and measurement that no margin of error captures (Biemer & Lyberg, 2003).
Disclosure in practice
Most news reports cannot include every item, so standards usually require a short statement with the report and full details available elsewhere, for example on the research company's website. A typical short statement reads: "The survey was conducted online by Example Research for the Daily News among 1,512 adults aged 18 and over in Canada between 3 and 6 March. Results are weighted by age, sex, region and education. Full tables and question wording are available at example.com." Figure 2 shows how often such items are reported in news coverage of polls.
Journalists play a part as well. Guidance for reporters lists questions to ask before reporting any poll, covering most of the disclosure items (Gawiser & Witt, 2007). Research on the reporting of polls has long found that news coverage often omits important methodological details (Lavrakas & Traugott, 2000).
Transparency beyond the minimum
Some organisations go further. AAPOR's Transparency Initiative recognises organisations that commit to routine disclosure of their methods. Some publish full questionnaires, computer tables and even anonymised data sets. This reflects a wider movement towards openness in research, in which methods and data are shared so that results can be checked and reproduced (Nosek et al., 2015).
Enforcement
Professional bodies can investigate complaints that members failed to disclose methods, and may publicly censure those who refuse. Their reach is limited to members, and many published surveys come from organisations outside any association. In some countries, election laws require published pre-election polls to include certain information, and some restrict publication in the final days before a vote.
An example
A charity publishes a press release stating that "eight in ten parents worry about screen time". A journalist asks for the details and learns that the survey was an online questionnaire promoted on the charity's social media pages, completed by 640 self-selected parents, with no weighting. The question asked respondents whether they agreed that "too much screen time is harming children". The journalist reports the finding with these details, noting that the sample was self-selected and the question was worded in one direction (illustrative example).
Conclusion
Disclosure standards ensure that published survey results come with the information needed to judge them: who did the survey, whom it represents, how respondents were chosen, what was asked, when, how and with what precision. Following them is a mark of professional research, and asking for them is the best protection readers have against misleading figures.
References
American Association for Public Opinion Research. (2021). AAPOR code of professional ethics and practices. AAPOR. https://aapor.org/standards-and-ethics/
American Association for Public Opinion Research. (2023). Standard definitions: Final dispositions of case codes and outcome rates for surveys (10th ed.). AAPOR. https://aapor.org/standards-and-ethics/standard-definitions/
Biemer, P. P., & Lyberg, L. E. (2003). Introduction to survey quality. Wiley. https://doi.org/10.1002/0471458740
British Polling Council. (n.d.). Officers, objects and rules. British Polling Council.
ESOMAR & WAPOR. (2014). ESOMAR/WAPOR guideline on opinion polls and published surveys. ESOMAR.
Gawiser, S. R., & Witt, G. E. (2007). 20 questions a journalist should ask about poll results (3rd ed.). National Council on Public Polls.
International Organization for Standardization. (2019). Market, opinion and social research, including insights and data analytics — Vocabulary and service requirements (ISO Standard No. 20252:2019).
Kennedy, C., Blumenthal, M., Clement, S., Clinton, J. D., Durand, C., Franklin, C., McGeeney, K., Miringoff, L., Olson, K., Rivers, D., Saad, L., Witt, G. E., & Wlezien, C. (2018). An evaluation of the 2016 election polls in the United States. Public Opinion Quarterly, 82(1), 1–33. https://doi.org/10.1093/poq/nfx047
Lavrakas, P. J., & Traugott, M. W. (Eds.). (2000). Election polls, the news media, and democracy. Chatham House.
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., . . . Yarkoni, T. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425. https://doi.org/10.1126/science.aab2374
Shirani-Mehr, H., Rothschild, D., Goel, S., & Gelman, A. (2018). Disentangling bias and variance in election polls. Journal of the American Statistical Association, 113(522), 607–614. https://doi.org/10.1080/01621459.2018.1448823
Sturgis, P., Kuha, J., Baker, N., Callegaro, M., Fisher, S., Green, J., Jennings, W., Lauderdale, B. E., & Smith, P. (2018). An assessment of the causes of the errors in the 2015 UK general election opinion polls. Journal of the Royal Statistical Society: Series A (Statistics in Society), 181(3), 757–781. https://doi.org/10.1111/rssa.12329