Inquiry · Inquiry in practice
Philosophical Inquiry Into Survey Questions: What Can Be Asked?
Not every question can be meaningfully asked of every person. This article draws on philosophy and survey research to examine the conditions a survey question must meet to produce a meaningful answer: a sound presupposition, a clear concept, an answer the respondent can know and a request it is fair to make.
A survey question looks simple: a sentence ending in a question mark, followed by some answer options. Behind it lie assumptions that are easy to overlook. The question assumes that the respondent understands it as intended, that there is a fact or attitude for them to report, that they can access it and that it is reasonable to ask. When any of these assumptions fails, the survey still records an answer, but the answer may mean little. Philosophy and survey research together help clarify what can, and cannot, be asked.
Every question has presuppositions
The philosopher R. G. Collingwood (1940) argued that every question rests on presuppositions, things that must be taken for granted for the question to arise at all. The question "Which brand of coffee do you prefer?" presupposes that the respondent drinks coffee and distinguishes between brands. If the presupposition is false, there is no correct answer, only an answer.
Presuppositions can also shape the answers they receive. Elizabeth Loftus and John Palmer (1974) showed people a film of a car accident and asked how fast the cars were going when they "hit" or "smashed into" each other. Those asked with the word "smashed" gave higher estimates of speed and, a week later, were more likely to report having seen broken glass, though there was none in the film. The wording of a question carried information that changed what people remembered.
Is there anything there to report?
A survey treats answers as reports of something that already exists in the respondent's mind. Philip Converse (1964) questioned this assumption. Studying panel surveys in which the same people were asked the same policy questions several times, he found that many answers changed from one interview to the next in ways that looked close to random. He concluded that many respondents held non-attitudes: they had no real opinion, but produced one when asked. Bishop et al. (1980) went further, asking people about a fictitious "Public Affairs Act". About a third offered an opinion on a law that did not exist.
John Zaller and Stanley Feldman (1992) proposed a more moderate view. People do not usually hold fixed answers to survey questions; instead, they hold a mix of considerations, and they construct an answer on the spot from whichever considerations are most accessible at that moment. The answer is real, but it depends on the context in which the question is asked. On this view, a survey does not simply read off an opinion stored in advance. It samples from a range of possible answers, and the questions asked just before, the wording used and recent news can all change which answer is produced.
Can the respondent know the answer?
Some questions ask for knowledge that people do not have. Richard Nisbett and Timothy Wilson (1977) reviewed evidence that people are often unable to report accurately on the causes of their own behaviour. When asked why they made a choice, they give plausible reasons, but these frequently do not match the factors that actually influenced them. Questions that ask "why" therefore need careful interpretation. They reveal the reasons people find convincing, which may differ from the reasons that moved them.
Roger Tourangeau, Lance Rips and Kenneth Rasinski (2000) describe answering a survey question as four tasks: understanding the question, retrieving relevant information from memory, forming a judgment and selecting a response. A question can fail at any stage, for example if it asks about events too far in the past to recall or requires an estimate the respondent has no basis to make.
Is the concept clear?
A question measures a concept, and the concept must be clear enough to measure. Lee Cronbach and Paul Meehl (1955) argued that the meaning of a measured concept, which they called a construct, is defined by its place in a network of relationships with other concepts and observations. A question about "customer loyalty", for instance, assumes there is a single thing called loyalty, related in predictable ways to repeat purchase, recommendation and resistance to competitors. If the concept is vague, answers cannot be interpreted.
Respondents read questions as conversation
The philosopher Paul Grice (1975) argued that conversation follows cooperative principles: speakers assume that what others say is relevant, truthful, sufficiently informative and clear. Norbert Schwarz (1999) showed that survey respondents apply the same assumptions to questionnaires. They treat every feature of the question, including the answer scale, as meaningful. In one study he describes, people asked how successful they had been in life were far more likely to choose a low value when the scale ran from 0 to 10 than when it ran from −5 to +5. The numbers suggested different meanings for the lower end: an absence of success in one case, explicit failure in the other.
Schuman and Presser (1981) documented many such effects, showing that changes in wording, order and response options can shift answers substantially. These are not mere technical flaws. They show that respondents are interpreting the question, as any listener would.
Is it fair to ask?
Finally, some questions raise ethical and practical issues. Tourangeau and Yan (2007) reviewed research on sensitive questions, such as those about income, illegal behaviour or health, and found that such questions produce higher rates of refusal and misreporting, especially when asked by an interviewer rather than in a self-administered form. A question may be technically answerable yet still intrusive, or likely to produce distorted answers.
Figure 1 summarises these conditions.
Implications
These conditions suggest practical checks. Test whether the presuppositions of a question hold for every respondent, and route around them when they do not. Offer a "don't know" or "no opinion" option where non-attitudes are likely. Ask about behaviour and experience rather than causes where possible. Define concepts clearly and pre-test questions with people from the target group. Read every question as a respondent would, as a statement in a conversation, and consider what it implies. A survey can only be as meaningful as the questions it asks, and a good question begins with clear thinking about what can be asked at all.
References
Bishop, G. F., Oldendick, R. W., Tuchfarber, A. J., & Bennett, S. E. (1980). Pseudo-opinions on public affairs. Public Opinion Quarterly, 44(2), 198–209. https://doi.org/10.1086/268584
Collingwood, R. G. (1940). An essay on metaphysics. Clarendon Press.
Converse, P. E. (1964). The nature of belief systems in mass publics. In D. E. Apter (Ed.), Ideology and discontent (pp. 206–261). Free Press.
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957
Grice, H. P. (1975). Logic and conversation. In P. Cole & J. L. Morgan (Eds.), Syntax and semantics: Vol. 3. Speech acts (pp. 41–58). Academic Press.
Loftus, E. F., & Palmer, J. C. (1974). Reconstruction of automobile destruction: An example of the interaction between language and memory. Journal of Verbal Learning and Verbal Behavior, 13(5), 585–589. https://doi.org/10.1016/S0022-5371(74)80011-3
Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231–259. https://doi.org/10.1037/0033-295X.84.3.231
Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. Academic Press.
Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54(2), 93–105. https://doi.org/10.1037/0003-066X.54.2.93
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press. https://doi.org/10.1017/CBO9780511819322
Tourangeau, R., & Yan, T. (2007). Sensitive questions in surveys. Psychological Bulletin, 133(5), 859–883. https://doi.org/10.1037/0033-2909.133.5.859
Zaller, J., & Feldman, S. (1992). A simple theory of the survey response: Answering questions versus revealing preferences. American Journal of Political Science, 36(3), 579–616. https://doi.org/10.2307/2111583