Skip to content

Surveys · Questionnaire design

The Epistemology of Questionnaire Design: Shaping Knowledge Generation in Mass Surveys

Surveys do not simply record what people think; the questions they ask help to define what can be known. This article examines how questionnaire design shapes the knowledge surveys produce, the main critiques of survey knowledge and what follows for good practice.

ModQu research article · 1,613 words · 20 references (APA 7th edition)

Download PDF

A survey result such as "62 per cent of customers are satisfied" looks like a simple fact about the world. But that figure exists only because someone decided to ask about satisfaction, chose particular words, offered particular answer categories, asked particular people and added up their answers in a particular way. Each of these decisions shapes what the survey can find. Epistemology is the study of knowledge: what it is, how it is produced and how far it can be trusted. Asking epistemological questions about questionnaires means asking what kind of knowledge surveys produce and how the design of the questionnaire shapes it.

This article examines how questionnaire design shapes knowledge, reviews the main critiques of survey knowledge and draws out what they mean for designing and interpreting surveys.

From concept to number

Every survey measure is the end of a chain of decisions. A researcher starts with a concept, such as trust, satisfaction or social class, which cannot be observed directly. The concept is turned into one or more questions, the questions offer answer categories, respondents interpret and answer them, and their answers are coded and combined into statistics.

A flow of six steps from concept to published number. Concept: deciding what is worth knowing and how to define it. Question: choosing words, frame and reference period. Answer options: deciding which answers are possible. Respondent: interpreting the question and forming an answer. Coding and analysis: grouping, scoring and weighting answers. Published number: the statistic that becomes a fact.
Figure 1. From concept to published number: each step involves choices that shape what the survey can know. Diagram by ModQu.

At each step, choices are made that could have been made differently. The concept could have been defined differently; the question could have used other words; the categories could have been drawn in other places. The result is not a direct observation of the concept but a measurement shaped by these choices. The basic idea of measurement as the assignment of numbers to objects according to rules makes this explicit: the rules matter as much as the objects (Stevens, 1946).

How questions shape answers

Answers depend on the question

Decades of experimental research have shown that apparently minor changes in wording, order, format and context can change answers substantially (Schuman & Presser, 1981). These effects are not just errors to be eliminated. They show that, for many topics, respondents do not hold a fixed answer waiting to be recorded. Instead, they construct an answer at the moment of asking, using whatever information the question and its context make available (Schwarz, 1999; Tourangeau et al., 2000).

Zaller and Feldman (1992) argued that most people hold a mix of partly conflicting considerations about political issues, and that a survey answer reflects the considerations that happen to be most accessible when the question is asked. On this view, a survey does not reveal a pre-existing opinion so much as sample from a range of possible answers, with the question influencing which part of the range is sampled.

Nonattitudes

Converse (1964) found that many respondents in panel surveys gave answers to policy questions that varied almost randomly over time, and concluded that many people did not hold real attitudes on these issues. They answered because they were asked. Questions that do not offer a "don't know" option, or that ask about obscure topics, can therefore produce opinions that exist only in the survey. Later work showed that people will even give opinions about fictitious laws (Bishop, 2005).

Categories create classifications

Answer categories determine which distinctions a survey can see. A question about ethnicity, gender, occupation or household type that offers a fixed list of categories makes those categories the units of analysis. Respondents who do not fit must choose the nearest option or none. Over time, categories used in official surveys can shape how people understand themselves and how society is described (Desrosières, 1998).

Reference periods and memory

Questions about behaviour seem more factual than questions about opinion, but they too depend on design. A question about how often someone visited a doctor "in the last 12 months" asks respondents to recall, date and count events, and the answer depends on how they do so. People often place events inside the reference period when they actually happened earlier, and they may estimate rather than count when events are frequent (Tourangeau et al., 2000). The length of the reference period, the examples given and the order of questions all influence the number that results. Even apparently simple facts reach the survey through the respondent's memory and judgement.

Two views of what a survey measures

It is useful to distinguish two broad views of survey measurement. In the first, which can be called the realist view, each respondent has a true value, such as a true attitude or a true number of doctor visits, and the survey aims to record it with as little error as possible. Differences caused by wording or order are errors to be minimised. In the second, the constructivist view, answers are produced in the interaction between the respondent and the questionnaire, so that there is no single true value independent of how the question is asked.

Most survey methodologists take a position between these views. For factual questions, a true value usually exists, and the task is to design questions that help respondents report it. For attitudes, the constructivist view often fits better, and the task is to understand how answers depend on the question and to report results with that dependence in mind (Schwarz, 1999).

Critiques of survey knowledge

Does public opinion exist?

Some of the sharpest critiques came early. Blumer (1948) argued that opinion polling treated society as a collection of separate individuals with equal weight, ignoring the fact that public opinion is formed through organised groups, influence and debate. Bourdieu (1979) went further, arguing that "public opinion" as measured by polls was an artefact: polls assume that everyone has an opinion, that all opinions count equally and that there is agreement about which questions are worth asking. None of these assumptions, he argued, is true.

Historians have shown how polling changed public life. Herbst (1993) traced how numerical measures of opinion displaced earlier forms of public expression, such as petitions, crowds and letters. Igo (2007) showed how surveys taught Americans to think of themselves as members of a statistical public, with "average" views and behaviours. Osborne and Rose (1999) argued that opinion research did not simply discover public opinion but helped to create it as a phenomenon that could be measured, reported and acted on.

The interview as an interaction

Another line of critique focused on the interview itself. Cicourel (1964) argued that standardised interviewing ignored the ways in which meaning is negotiated between people. Studies of recorded interviews showed that respondents frequently misunderstood questions, and that the rules of standardised interviewing prevented interviewers from clarifying them (Suchman & Jordan, 1990). Briggs (1986) argued that the interview format imposes the researcher's way of communicating on respondents whose own ways of talking about a subject may be quite different.

What follows for survey design

These critiques do not mean that surveys produce no knowledge. They mean that survey knowledge is constructed, conditional and partial, and that its construction should be visible. Several practical principles follow.

Make the concept explicit

A survey should start with a clear definition of what it aims to measure and why. Without one, there is no way to judge whether the questions measure it (Saris & Gallhofer, 2014).

Ground questions in respondents' meanings

Qualitative work, cognitive interviews and pretests help to ensure that questions use concepts and words that respondents recognise. This reduces the risk that the survey imposes categories that do not fit the people it describes.

Allow for the absence of opinion

Offering "don't know" or "no opinion" options, or asking whether respondents have thought about an issue before asking their view, helps to separate real attitudes from answers produced by the survey.

Test the sensitivity of results

Where results may depend on wording, split-ballot experiments that compare versions show how much. A finding that holds across several wordings is more trustworthy than one that depends on a single question (Schuman & Scott, 1987).

Report how numbers were made

Publishing the exact question wording, the answer options, the sample and the methods allows readers to judge what a statistic means. A result stated without its question is a number without its meaning.

Surveys and other kinds of data

Large amounts of data are now generated by digital activity, and some have argued that such data make theory-driven measurement unnecessary. Critics point out that all data are produced through choices about what to record and how, so that the epistemological questions raised about surveys apply equally to new data sources (Kitchin, 2014). Surveys have one advantage here: because the questions are designed deliberately, the choices that shape the data can be documented and examined.

In business surveys

For organisations that use survey results to make decisions, these ideas have a practical meaning. A satisfaction score, a brand rating or a market estimate reflects the questions that produced it. Two surveys of the same customers that ask differently may give different answers, and neither is simply "the truth". The best protection is to understand how each measure was constructed, to compare results across methods where possible and to treat any single number as an estimate that depends on its design (Groves et al., 2009).

Conclusion

Questionnaire design is not a neutral technical exercise that captures what already exists. It decides which concepts count, which answers are possible and which distinctions can be seen. Critiques from sociology, history and cognitive psychology have shown how survey knowledge is constructed and how it can mislead. The response is not to abandon surveys but to design them with awareness of these choices, ground them in the meanings of respondents, test their sensitivity and report openly how their numbers were made. Surveys generate knowledge, and good design makes that knowledge more honest about its own foundations.

References

Bishop, G. F. (2005). The illusion of public opinion: Fact and artifact in American public opinion polls. Rowman & Littlefield.

Blumer, H. (1948). Public opinion and public opinion polling. American Sociological Review, 13(5), 542–549.

Bourdieu, P. (1979). Public opinion does not exist. In A. Mattelart & S. Siegelaub (Eds.), Communication and class struggle: Vol. 1. Capitalism, imperialism (pp. 124–130). International General.

Briggs, C. L. (1986). Learning how to ask: A sociolinguistic appraisal of the role of the interview in social science research. Cambridge University Press.

Cicourel, A. V. (1964). Method and measurement in sociology. Free Press.

Converse, P. E. (1964). The nature of belief systems in mass publics. In D. E. Apter (Ed.), Ideology and discontent (pp. 206–261). Free Press.

Desrosières, A. (1998). The politics of large numbers: A history of statistical reasoning (C. Naish, Trans.). Harvard University Press.

Groves, R. M., Fowler, F. J., Jr., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology (2nd ed.). Wiley.

Herbst, S. (1993). Numbered voices: How opinion polling has shaped American politics. University of Chicago Press.

Igo, S. E. (2007). The averaged American: Surveys, citizens, and the making of a mass public. Harvard University Press.

Kitchin, R. (2014). Big Data, new epistemologies and paradigm shifts. Big Data & Society, 1(1). https://doi.org/10.1177/2053951714528481

Osborne, T., & Rose, N. (1999). Do the social sciences create phenomena? The example of public opinion research. British Journal of Sociology, 50(3), 367–396. https://doi.org/10.1111/j.1468-4446.1999.00367.x

Saris, W. E., & Gallhofer, I. N. (2014). Design, evaluation, and analysis of questionnaires for survey research (2nd ed.). Wiley. https://doi.org/10.1002/9781118634646

Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. Academic Press.

Schuman, H., & Scott, J. (1987). Problems in the use of survey questions to measure public opinion. Science, 236(4804), 957–959. https://doi.org/10.1126/science.236.4804.957

Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54(2), 93–105. https://doi.org/10.1037/0003-066X.54.2.93

Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103(2684), 677–680. https://doi.org/10.1126/science.103.2684.677

Suchman, L., & Jordan, B. (1990). Interactional troubles in face-to-face survey interviews. Journal of the American Statistical Association, 85(409), 232–241. https://doi.org/10.1080/01621459.1990.10475331

Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press. https://doi.org/10.1017/CBO9780511819322

Zaller, J., & Feldman, S. (1992). A simple theory of the survey response: Answering questions versus revealing preferences. American Journal of Political Science, 36(3), 579–616. https://doi.org/10.2307/2111583