Skip to content

Surveys · Questionnaire design

Leading and Loaded Questions: How Wording Biases Answers

A single word or phrase can shift survey results by many percentage points. This article explains how leading and loaded questions work, what the classic experiments show, and how to write questions that let respondents' own views come through.

ModQu research article · 1,151 words · 12 references (APA 7th edition)

Download PDF

Survey questions are supposed to discover what people think, not tell them what to think. Yet the wording of a question can change the answers it receives, sometimes dramatically. When the wording pushes respondents towards a particular answer, the question is called leading. When it uses words with strong emotional or political associations, it is called loaded. Both produce results that reflect the question as much as the respondents. This article explains how these effects work, describes some of the best-known evidence, and sets out practical ways to avoid them.

Why wording matters so much

Answering a survey question is not a matter of reading off a fixed opinion stored in memory. For many topics, especially those people have not thought about much, respondents build an answer on the spot from whatever considerations come to mind when the question is asked (Zaller & Feldman, 1992). The question itself is one of the strongest influences on what comes to mind. Respondents also assume that the questioner is being cooperative and informative, so they treat every word, example and response option as a clue to what is being asked and what answer is expected (Schwarz, 1999). A question that signals a preferred answer is therefore likely to receive it.

Classic evidence

Some of the earliest experiments in survey research tested the effect of single words. In one, Americans were asked whether the government should "forbid" or should "not allow" public speeches against democracy. The two versions describe the same policy, yet many more respondents agreed that such speeches should be "not allowed" than agreed that they should be "forbidden" (Rugg, 1941). The word "forbid" sounds harsher, and some respondents who were willing to withhold permission balked at an outright ban. The finding has been replicated decades later (Schuman & Presser, 1981).

A second classic concerns public spending. When Americans were asked whether the government spends too much, too little or about the right amount on "welfare," large proportions said too much. When the same programmes were described as "assistance to the poor," most said too little (Smith, 1987; Rasinski, 1989). The difference arises from the associations the words carry, not their literal meaning.

Wording affects memory as well as opinion. In a famous experiment, people who had watched a film of a traffic accident estimated higher speeds when asked how fast the cars were going when they "smashed into" each other than when asked about cars that "hit" each other. A week later, those who had heard "smashed" were also more likely to report having seen broken glass, though there was none in the film (Loftus & Palmer, 1974).

Framing works in a similar way. Tversky and Kahneman (1981) showed that people choose differently between the same two programmes when outcomes are described in terms of lives saved rather than lives lost. Political scientists have since documented how the frames used to describe an issue, such as describing a rally as a free speech issue or a public safety issue, shift opinion (Chong & Druckman, 2007).

Common forms of leading and loaded wording

Six forms of biased wording with examples. Loaded words: welfare versus assistance to the poor. Leading stems: Don't you agree that. Presupposition: How much has our new service improved your experience? Prestige cues: Experts recommend; do you support? One-sided questions: Do you support the plan? without mentioning opposing. Unbalanced options: Good, Very good, Excellent.
Figure 1. Six common forms of leading and loaded wording. Examples written by ModQu, drawing on Payne (1951), Schuman and Presser (1981) and Bradburn et al. (2004).

Biased wording takes several recognisable forms (Payne, 1951; Bradburn et al., 2004):

Agree-disagree statements are a special case. Some respondents tend to agree with whatever statement is put to them, a tendency called acquiescence, so the direction in which a statement is worded affects the results. Questions that ask directly about the attribute, with options specific to it, generally produce better data than agree-disagree items (Saris et al., 2010).

How to write neutral questions

Avoiding leading and loaded wording does not require special talent. It requires discipline and testing:

  1. Use plain, descriptive language. Describe a policy, product or behaviour in the most neutral terms available, and prefer concrete descriptions to labels.
  2. Present both sides. Ask whether respondents "favour or oppose," find something "easy or difficult," or are "satisfied or dissatisfied."
  3. Balance the response options. Give equal numbers of positive and negative options with symmetrical labels.
  4. Remove assumptions. Ask first whether something happened before asking how much it helped.
  5. Leave out endorsements. Unless the purpose is to test the effect of an endorsement, do not mention who supports a position.
  6. Have someone else read it. People who do not share the author's view are good at spotting loaded language. Cognitive interviews with members of the target population are even better.
  7. Test alternatives experimentally. When a question matters and wording choices are uncertain, randomly assign respondents to different versions. The difference between versions measures the wording effect directly (Schuman & Presser, 1981).

When wording effects are the point

Not every wording effect is a problem. Researchers and businesses often want to know how people respond to different descriptions, for example which explanation of a price change customers find most acceptable, or which name for a product suggests the right qualities. In such cases the wording is the treatment, and randomised experiments with alternative wordings are exactly the right tool. The problem arises only when a single, slanted wording is presented as if it measured an underlying opinion.

Why it matters for business decisions

Biased questions are tempting because they produce pleasing numbers. A customer survey that asks "How much did you enjoy your visit?" will yield higher scores than one that asks "How would you rate your visit?" A product team that asks "Would this feature make your work easier?" will hear more "yes" answers than one that asks how useful the feature would be. The cost appears later, when decisions based on inflated results fail to deliver. Neutral questions may produce less flattering numbers, but they are numbers that can be trusted, compared over time and acted on.

Conclusion

Leading and loaded questions bias answers because respondents take cues from the question about what to think and what answer is expected. Decades of experiments show that single words, framings and response options can shift results substantially. The remedy is straightforward: describe things neutrally, present both sides, balance the options, remove assumptions and test wording before relying on it. Questions written this way do not guarantee accurate answers, but they remove one of the most avoidable sources of error in survey research.

References

Bradburn, N. M., Sudman, S., & Wansink, B. (2004). Asking questions: The definitive guide to questionnaire design for market research, political polls, and social and health questionnaires (Rev. ed.). Jossey-Bass.

Chong, D., & Druckman, J. N. (2007). Framing theory. Annual Review of Political Science, 10, 103–126. https://doi.org/10.1146/annurev.polisci.10.072805.103054

Loftus, E. F., & Palmer, J. C. (1974). Reconstruction of automobile destruction: An example of the interaction between language and memory. Journal of Verbal Learning and Verbal Behavior, 13(5), 585–589. https://doi.org/10.1016/S0022-5371(74)80011-3

Payne, S. L. (1951). The art of asking questions. Princeton University Press.

Rasinski, K. A. (1989). The effect of question wording on public support for government spending. Public Opinion Quarterly, 53(3), 388–394. https://doi.org/10.1086/269158

Rugg, D. (1941). Experiments in wording questions: II. Public Opinion Quarterly, 5(1), 91–92. https://doi.org/10.1086/265467

Saris, W. E., Revilla, M., Krosnick, J. A., & Shaeffer, E. M. (2010). Comparing questions with agree/disagree response options to questions with item-specific response options. Survey Research Methods, 4(1), 61–79. https://doi.org/10.18148/srm/2010.v4i1.2682

Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. Academic Press.

Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54(2), 93–105. https://doi.org/10.1037/0003-066X.54.2.93

Smith, T. W. (1987). That which we call welfare by any other name would smell sweeter: An analysis of the impact of question wording on response patterns. Public Opinion Quarterly, 51(1), 75–83. https://doi.org/10.1086/269015

Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211(4481), 453–458. https://doi.org/10.1126/science.7455683

Zaller, J., & Feldman, S. (1992). A simple theory of the survey response: Answering questions versus revealing preferences. American Journal of Political Science, 36(3), 579–616. https://doi.org/10.2307/2111583