Inquiry · The art of the question
Question Design for Exploration Rather Than Confirmation
Many research questions are designed, often without anyone noticing, to confirm what the asker already believes. This article explains why that happens, what the research on hypothesis testing and survey design shows, and how to write questions that explore rather than confirm.
A product manager who believes customers want a new feature asks them, "Would you find it useful if the app could do X?" Most say yes. The feature is built, and few people use it. The research did not fail because the customers lied. It failed because the question was designed, without anyone intending it, to confirm a belief rather than to test it. Almost any proposal sounds useful when described in the abstract, and people are inclined to agree with the person asking.
This pattern is one of the most common and costly failures in inquiry. It appears in market research, management decisions, scientific studies and everyday conversations. This article draws on research in cognitive psychology, survey methodology and the philosophy of science to explain why confirming questions come so naturally, and how to design questions that explore what is actually the case.
Exploration and confirmation
Researchers often distinguish two modes of inquiry. Confirmatory research tests a specific hypothesis that was stated in advance. Exploratory research seeks to discover what is there, without committing in advance to what will be found. The statistician John Tukey (1977) championed exploratory data analysis, in which the analyst looks at data from many angles to see what patterns emerge, and later argued that science needs both modes: exploration to generate ideas worth testing, and confirmation to test them rigorously (Tukey, 1980). Robert Stebbins (2001) made a similar case for exploratory research in the social sciences, describing it as a broad, flexible search for understanding of an area about which little is known.
The two modes need different kinds of questions. Problems arise when a question that should be exploratory is framed as confirmatory, so that it can only find what the researcher already expected, or when an exploratory finding is presented as if it were a confirmed result. This article is mainly about the first problem.
Why we ask confirming questions
The positive test strategy
The tendency to ask confirming questions is deeply rooted. Peter Wason (1960) gave people the sequence 2, 4, 6 and asked them to discover the rule that generated it by proposing other sequences and being told whether each fitted. Most people formed a hypothesis such as "numbers increasing by two" and tested sequences that fitted it, such as 10, 12, 14. Every test returned yes. Many then announced their rule with confidence, only to learn it was wrong: the actual rule was any increasing sequence. To discover this, they would have needed to test a sequence that their hypothesis said should fail, such as 1, 2, 3.
Joshua Klayman and Young-Won Ha (1987) called this the positive test strategy: testing cases that one expects to fit one's hypothesis. They showed that it is often a reasonable heuristic, but that it systematically fails when the true answer is broader than the hypothesis. Raymond Nickerson (1998), reviewing the research, concluded that confirmation bias, the tendency to seek and interpret evidence in ways that favour existing beliefs, is one of the most pervasive problems in human reasoning.
Confirming questions in conversation
The same tendency shows up when people ask other people questions. Mark Snyder and William Swann (1978) asked participants to interview someone to find out whether that person was an extravert. The interviewers mostly chose questions that would reveal extraversion, such as "What would you do if you wanted to liven things up at a party?" When asked to find out whether the person was an introvert, they chose questions such as "What factors make it hard for you to really open up to people?" The answers tended to confirm whichever hypothesis the interviewer had been given, regardless of the person's actual personality, because almost anyone can describe how they would liven up a party or what makes it hard to open up.
Trope and Bassok (1982) showed that people are capable of asking more diagnostic questions, ones whose answers would differ depending on whether the hypothesis is true, when they are made aware of the need to. Jonathan Nelson (2005) analysed several formal measures of how useful a question is, all of which favour questions whose possible answers would most change the asker's beliefs. A question whose answer is predictable regardless of the truth is, by these measures, nearly worthless.
Agreement and acquiescence
Confirming questions are made worse by the tendency of respondents to agree. Jon Krosnick (1999) reviewed evidence that many respondents agree with statements in surveys regardless of their content, a pattern known as acquiescence. Zuckerman et al. (1995) showed that this combines with the positive test strategy: when an interviewer asks a question that assumes a hypothesis, and the respondent tends to agree, the hypothesis is "confirmed" twice over. A product team that asks "Would this feature be useful?" is relying on exactly this combination.
Fixed answer options
Closed questions add another layer. Howard Schuman and Stanley Presser (1979) showed that when people are given a list of possible answers, they tend to choose from the list, and answers that are not on it largely disappear. A survey that asks customers to choose their main reason for leaving from a list of five reasons written by the marketing team can only confirm one of those five. If the real reason is a sixth, the survey will not find it.
Principles for exploratory questions
The research suggests several principles for designing questions that explore rather than confirm.
1. Ask about behaviour, not hypotheticals
Questions about hypothetical futures ("Would you use…?", "Would you pay…?") invite agreement and are poor predictors of what people actually do. Questions about specific past behaviour ("The last time you needed to do X, what did you do?") are harder to answer to please the questioner, and reveal whether the problem the researcher has in mind actually exists. Christensen et al. (2016) argue that understanding the "job" a customer is trying to get done, by investigating the circumstances in which they chose or abandoned a product, is more useful than asking about preferences in the abstract.
2. Start open and stay open longer than feels comfortable
Exploratory interviews should begin with broad, open questions that do not presuppose the answer. The anthropologist James Spradley (1979) recommended "grand tour" questions, such as "Could you describe a typical day at work?", which invite the respondent to set the agenda before the interviewer narrows it. Kvale and Brinkmann (2009) recommend that the interviewer resist the urge to move quickly to the topics they are interested in, because the most valuable information often comes from the parts of an account the interviewer did not anticipate.
3. Remove the hypothesis from the question
A question that contains the hypothesis tends to confirm it. "Did the long wait make you frustrated?" assumes both the wait and the frustration. "How did you find the process of getting help?" does not. Rewriting a question to remove the hypothesis is one of the simplest and most effective improvements a researcher can make. Figure 1 gives some examples.
4. Ask questions whose answers could surprise you
A useful test for any question is to ask, before collecting the data, what answer would surprise the team and what it would do if it got that answer. If no answer would be surprising, or if every possible answer would lead to the same decision, the question is not exploring anything. Nelson's (2005) analysis implies the same practical rule: the most useful questions are those with the greatest uncertainty about the answer.
5. Consider several hypotheses at once
The geologist Thomas Chamberlin (1890) recommended the "method of multiple working hypotheses": rather than adopting one explanation and seeking evidence for it, the inquirer should keep several explanations in view and seek evidence that distinguishes between them. John Platt (1964) argued that fields that followed this approach, which he called strong inference, advanced faster than fields that did not. In practice, this means listing the possible explanations for a pattern before designing questions, and choosing questions whose answers would differ between explanations. Lord et al. (1984) found that instructing people to "consider the opposite" reduced biased evaluation of evidence more effectively than simply asking them to be fair.
6. Let the categories emerge
Instead of writing answer options in advance, exploratory research can let the categories come from the data. Glaser and Strauss (1967), in developing grounded theory, argued that researchers should build concepts from the data rather than forcing data into pre-existing categories. In a survey, an open question asked of a pilot sample can supply the answer options for a later closed question, so that the closed question includes the answers people actually give.
7. Explore the data, not only the respondents
Exploration applies to analysis as well as to asking. Tukey (1977) argued that analysts should look at data in many ways, through plots, summaries and breakdowns, before deciding what model to fit, because the most important features of a data set are often ones nobody thought to test for. John Behrens (1997) described exploratory data analysis as a stance as much as a set of techniques: an attitude of openness to what the data show, combined with scepticism about patterns that may be accidents. In practice, this means plotting the distribution of every key variable before averaging it, looking at the cases that do not fit the pattern as well as those that do, and asking what the data would look like if the favoured explanation were wrong. Outliers and anomalies are often where exploration finds something new, but only if the analyst is looking for surprises rather than for confirmation.
Why organisations prefer confirming questions
Individual habits are only part of the problem. Organisations often create pressure for confirming questions. Research is frequently commissioned after a decision has effectively been made, to support it rather than to test it. A team that has invested months in a product is understandably reluctant to ask questions that might show the investment was misguided. Research budgets are often justified by promising specific answers, which encourages narrow, confirmatory designs. And findings that confirm a senior leader's view are easier to present than findings that challenge it.
Thomas Wedell-Wedellsborg (2017) found that managers frequently move straight from a problem as first stated to solutions, without questioning the framing, and that teams which deliberately reframe problems before solving them reach better outcomes. Making room for exploratory questions therefore requires organisational as well as individual changes: commissioning research before decisions are made, rewarding findings that change plans rather than only those that confirm them, and treating "we were wrong about this" as a valuable result.
The philosopher Karl Popper (1959) argued that scientific theories can never be proved by confirming instances, however many, but can be refuted by a single clear counter-instance, and that the mark of a scientific attitude is to seek out the tests most likely to show a theory false. Organisations rarely adopt such a stance in full, but even a modest version, such as asking in every research brief what result would show the current plan to be mistaken, shifts the balance from confirmation towards exploration.
Keeping exploration and confirmation separate
Exploration generates ideas; it does not prove them. A pattern discovered by looking at data from many angles may be real, or it may be a coincidence that would not appear in new data. Simmons et al. (2011) showed how flexible analysis can make chance findings look significant, and Norbert Kerr (1998) described the problem of "HARKing", hypothesising after the results are known, in which an exploratory finding is presented as if it had been predicted in advance.
The solution is not to avoid exploration but to label it honestly and to confirm important findings with new data. Nosek et al. (2018) describe pre-registration, in which researchers record their hypotheses and analysis plans before collecting data, as a way of keeping the two modes distinct: whatever was planned is confirmatory, and anything else is exploratory and should be treated as a lead to follow up. Edmondson and McManus (2007) make a related argument for organisational research, recommending that the choice between exploratory and confirmatory methods should fit how much is already known about a topic.
A worked example
A hypothetical bank wants to know why small business customers are moving to a competitor. The first draft of its survey asks, "How important were each of the following in your decision to move: lower fees, better online banking, better customer service?", with a five-point scale for each.
The draft has several confirming features. It assumes the customer made a deliberate decision. It lists three reasons chosen by the bank. And it asks about importance in the abstract, which invites people to rate everything as fairly important.
The exploratory redesign starts with interviews. Twelve former customers are asked to describe how they came to open an account with the competitor: what was happening in their business at the time, who was involved and what they did first. Several describe the same pattern. They applied for a loan, the bank took weeks to respond, and an accountant recommended the competitor, which offered a decision within days. Fees and online banking were mentioned, but as afterthoughts.
The survey that follows asks former customers, "What prompted you to look at other banks?" with options derived from the interviews, including "a loan or credit application", in random order and with an "other" option. A large share select the loan application. The bank's response, to speed up its lending decisions for small businesses, is one its original survey would never have suggested.
The bank then treats this exploratory finding as a hypothesis to be confirmed rather than a settled fact. It checks its own records and finds that business customers who applied for credit and waited more than two weeks for a decision were far more likely to close their accounts within a year than those who received a quick decision. It then pilots a faster lending process in two regions and compares account closures with regions that kept the old process. Exploration found the question; confirmation tested the answer.
A checklist for exploratory question design
Before fielding a survey, starting a set of interviews or commissioning an analysis, a team can run through a short checklist:
- Does any question contain the answer we expect, or a word that suggests it?
- Are we asking about real past behaviour, or about hypothetical intentions and abstract importance?
- Have we listed at least three possible explanations, and will our questions distinguish between them?
- Did the answer options come from respondents, or only from our own assumptions? Is there an "other" option?
- What result would surprise us, and what would we do if we got it?
- If every possible answer leads to the same decision, why are we asking?
- Have we said in advance which findings will be treated as confirmed and which as leads to follow up?
A question that passes these checks will not guarantee a discovery, but it gives the world a fair chance to show the researcher something new.
Conclusion
Confirming questions come naturally because people seek evidence that fits their beliefs, respondents tend to agree and fixed answer lists limit what can be found. Exploratory questions require deliberate design: asking about real behaviour, starting open, removing the hypothesis from the wording, seeking answers that could surprise, considering several explanations and letting categories emerge from the data. Exploration and confirmation both have a place, but only exploration can find what nobody expected, and in research that is often where the value lies.
References
Behrens, J. T. (1997). Principles and procedures of exploratory data analysis. Psychological Methods, 2(2), 131–160. https://doi.org/10.1037/1082-989X.2.2.131
Chamberlin, T. C. (1890). The method of multiple working hypotheses. Science, 15(366), 92–96. https://doi.org/10.1126/science.ns-15.366.92
Christensen, C. M., Hall, T., Dillon, K., & Duncan, D. S. (2016). Know your customers' "jobs to be done". Harvard Business Review, 94(9), 54–62.
Edmondson, A. C., & McManus, S. E. (2007). Methodological fit in management field research. Academy of Management Review, 32(4), 1246–1264. https://doi.org/10.5465/amr.2007.26586086
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine.
Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. https://doi.org/10.1207/s15327957pspr0203_4
Klayman, J., & Ha, Y.-W. (1987). Confirmation, disconfirmation, and information in hypothesis testing. Psychological Review, 94(2), 211–228. https://doi.org/10.1037/0033-295X.94.2.211
Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537–567. https://doi.org/10.1146/annurev.psych.50.1.537
Kvale, S., & Brinkmann, S. (2009). InterViews: Learning the craft of qualitative research interviewing (2nd ed.). Sage.
Lord, C. G., Lepper, M. R., & Preston, E. (1984). Considering the opposite: A corrective strategy for social judgment. Journal of Personality and Social Psychology, 47(6), 1231–1243. https://doi.org/10.1037/0022-3514.47.6.1231
Nelson, J. D. (2005). Finding useful questions: On Bayesian diagnosticity, probability, impact, and information gain. Psychological Review, 112(4), 979–999. https://doi.org/10.1037/0033-295X.112.4.979
Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. https://doi.org/10.1037/1089-2680.2.2.175
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114
Platt, J. R. (1964). Strong inference. Science, 146(3642), 347–353. https://doi.org/10.1126/science.146.3642.347
Popper, K. R. (1959). The logic of scientific discovery. Hutchinson.
Schuman, H., & Presser, S. (1979). The open and closed question. American Sociological Review, 44(5), 692–712. https://doi.org/10.2307/2094521
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
Snyder, M., & Swann, W. B., Jr. (1978). Hypothesis-testing processes in social interaction. Journal of Personality and Social Psychology, 36(11), 1202–1212. https://doi.org/10.1037/0022-3514.36.11.1202
Spradley, J. P. (1979). The ethnographic interview. Holt, Rinehart and Winston.
Stebbins, R. A. (2001). Exploratory research in the social sciences. Sage. https://doi.org/10.4135/9781412984249
Trope, Y., & Bassok, M. (1982). Confirmatory and diagnosing strategies in social information gathering. Journal of Personality and Social Psychology, 43(1), 22–34. https://doi.org/10.1037/0022-3514.43.1.22
Tukey, J. W. (1977). Exploratory data analysis. Addison-Wesley.
Tukey, J. W. (1980). We need both exploratory and confirmatory. The American Statistician, 34(1), 23–25. https://doi.org/10.1080/00031305.1980.10482706
Wason, P. C. (1960). On the failure to eliminate hypotheses in a conceptual task. Quarterly Journal of Experimental Psychology, 12(3), 129–140. https://doi.org/10.1080/17470216008416717
Wedell-Wedellsborg, T. (2017). Are you solving the right problems? Harvard Business Review, 95(1), 76–83.
Zuckerman, M., Knee, C. R., Hodgins, H. S., & Miyake, K. (1995). Hypothesis confirmation: The joint effect of positive test strategy and acquiescence response set. Journal of Personality and Social Psychology, 68(1), 52–60. https://doi.org/10.1037/0022-3514.68.1.52