Surveys · Questionnaire design
Leading and Loaded Questions: How Wording Biases Answers
A single word or phrase can shift survey results by many percentage points. This article explains how leading and loaded questions work, what the classic experiments show, and how to write questions that let respondents' own views come through.
Survey questions are supposed to discover what people think, not tell them what to think. Yet the wording of a question can change the answers it receives, sometimes dramatically. When the wording pushes respondents towards a particular answer, the question is called leading. When it uses words with strong emotional or political associations, it is called loaded. Both produce results that reflect the question as much as the respondents. This article explains how these effects work, describes some of the best-known evidence, and sets out practical ways to avoid them.
Why wording matters so much
Answering a survey question is not a matter of reading off a fixed opinion stored in memory. For many topics, especially those people have not thought about much, respondents build an answer on the spot from whatever considerations come to mind when the question is asked (Zaller & Feldman, 1992). The question itself is one of the strongest influences on what comes to mind. Respondents also assume that the questioner is being cooperative and informative, so they treat every word, example and response option as a clue to what is being asked and what answer is expected (Schwarz, 1999). A question that signals a preferred answer is therefore likely to receive it.
Classic evidence
Some of the earliest experiments in survey research tested the effect of single words. In one, Americans were asked whether the government should "forbid" or should "not allow" public speeches against democracy. The two versions describe the same policy, yet many more respondents agreed that such speeches should be "not allowed" than agreed that they should be "forbidden" (Rugg, 1941). The word "forbid" sounds harsher, and some respondents who were willing to withhold permission balked at an outright ban. The finding has been replicated decades later (Schuman & Presser, 1981).
A second classic concerns public spending. When Americans were asked whether the government spends too much, too little or about the right amount on "welfare," large proportions said too much. When the same programmes were described as "assistance to the poor," most said too little (Smith, 1987; Rasinski, 1989). The difference arises from the associations the words carry, not their literal meaning.
Wording affects memory as well as opinion. In a famous experiment, people who had watched a film of a traffic accident estimated higher speeds when asked how fast the cars were going when they "smashed into" each other than when asked about cars that "hit" each other. A week later, those who had heard "smashed" were also more likely to report having seen broken glass, though there was none in the film (Loftus & Palmer, 1974).
Framing works in a similar way. Tversky and Kahneman (1981) showed that people choose differently between the same two programmes when outcomes are described in terms of lives saved rather than lives lost. Political scientists have since documented how the frames used to describe an issue, such as describing a rally as a free speech issue or a public safety issue, shift opinion (Chong & Druckman, 2007).
Common forms of leading and loaded wording
Biased wording takes several recognisable forms (Payne, 1951; Bradburn et al., 2004):
- Loaded words. Terms with strong connotations, such as "bureaucrats," "handouts," "freedom" or "fair," carry an evaluation with them.
- Leading stems. Openings such as "Don't you agree that..." or "Wouldn't you say..." signal the expected answer.
- Presupposition. A question such as "How much has our new service improved your experience?" assumes that it has improved.
- Prestige and authority cues. Attaching a respected name or "most experts say" to one position invites agreement.
- One-sided questions. "Do you support the plan?" tends to draw more agreement than "Do you support or oppose the plan?"
- Unbalanced response options. A scale running from "good" to "excellent," with no negative options, pushes answers upwards.
Agree-disagree statements are a special case. Some respondents tend to agree with whatever statement is put to them, a tendency called acquiescence, so the direction in which a statement is worded affects the results. Questions that ask directly about the attribute, with options specific to it, generally produce better data than agree-disagree items (Saris et al., 2010).
How to write neutral questions
Avoiding leading and loaded wording does not require special talent. It requires discipline and testing:
- Use plain, descriptive language. Describe a policy, product or behaviour in the most neutral terms available, and prefer concrete descriptions to labels.
- Present both sides. Ask whether respondents "favour or oppose," find something "easy or difficult," or are "satisfied or dissatisfied."
- Balance the response options. Give equal numbers of positive and negative options with symmetrical labels.
- Remove assumptions. Ask first whether something happened before asking how much it helped.
- Leave out endorsements. Unless the purpose is to test the effect of an endorsement, do not mention who supports a position.
- Have someone else read it. People who do not share the author's view are good at spotting loaded language. Cognitive interviews with members of the target population are even better.
- Test alternatives experimentally. When a question matters and wording choices are uncertain, randomly assign respondents to different versions. The difference between versions measures the wording effect directly (Schuman & Presser, 1981).
When wording effects are the point
Not every wording effect is a problem. Researchers and businesses often want to know how people respond to different descriptions, for example which explanation of a price change customers find most acceptable, or which name for a product suggests the right qualities. In such cases the wording is the treatment, and randomised experiments with alternative wordings are exactly the right tool. The problem arises only when a single, slanted wording is presented as if it measured an underlying opinion.
Why it matters for business decisions
Biased questions are tempting because they produce pleasing numbers. A customer survey that asks "How much did you enjoy your visit?" will yield higher scores than one that asks "How would you rate your visit?" A product team that asks "Would this feature make your work easier?" will hear more "yes" answers than one that asks how useful the feature would be. The cost appears later, when decisions based on inflated results fail to deliver. Neutral questions may produce less flattering numbers, but they are numbers that can be trusted, compared over time and acted on.
Conclusion
Leading and loaded questions bias answers because respondents take cues from the question about what to think and what answer is expected. Decades of experiments show that single words, framings and response options can shift results substantially. The remedy is straightforward: describe things neutrally, present both sides, balance the options, remove assumptions and test wording before relying on it. Questions written this way do not guarantee accurate answers, but they remove one of the most avoidable sources of error in survey research.
References
Bradburn, N. M., Sudman, S., & Wansink, B. (2004). Asking questions: The definitive guide to questionnaire design for market research, political polls, and social and health questionnaires (Rev. ed.). Jossey-Bass.
Chong, D., & Druckman, J. N. (2007). Framing theory. Annual Review of Political Science, 10, 103–126. https://doi.org/10.1146/annurev.polisci.10.072805.103054
Loftus, E. F., & Palmer, J. C. (1974). Reconstruction of automobile destruction: An example of the interaction between language and memory. Journal of Verbal Learning and Verbal Behavior, 13(5), 585–589. https://doi.org/10.1016/S0022-5371(74)80011-3
Payne, S. L. (1951). The art of asking questions. Princeton University Press.
Rasinski, K. A. (1989). The effect of question wording on public support for government spending. Public Opinion Quarterly, 53(3), 388–394. https://doi.org/10.1086/269158
Rugg, D. (1941). Experiments in wording questions: II. Public Opinion Quarterly, 5(1), 91–92. https://doi.org/10.1086/265467
Saris, W. E., Revilla, M., Krosnick, J. A., & Shaeffer, E. M. (2010). Comparing questions with agree/disagree response options to questions with item-specific response options. Survey Research Methods, 4(1), 61–79. https://doi.org/10.18148/srm/2010.v4i1.2682
Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. Academic Press.
Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54(2), 93–105. https://doi.org/10.1037/0003-066X.54.2.93
Smith, T. W. (1987). That which we call welfare by any other name would smell sweeter: An analysis of the impact of question wording on response patterns. Public Opinion Quarterly, 51(1), 75–83. https://doi.org/10.1086/269015
Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211(4481), 453–458. https://doi.org/10.1126/science.7455683
Zaller, J., & Feldman, S. (1992). A simple theory of the survey response: Answering questions versus revealing preferences. American Journal of Political Science, 36(3), 579–616. https://doi.org/10.2307/2111583