Skip to content

The Generation of Knowledge · Trust, quality and the future

Modern Queries: Asking the Right Questions to Generate New Knowledge

Why the question comes before the answer in every inquiry, how a vague concern becomes an answerable research question, the costly error of answering the wrong question, and how good survey questions turn a research question into evidence.

ModQu research article · 1,218 words · 13 references (APA 7th edition)

Download PDF

Every piece of new knowledge begins with a question. The question decides what will be looked for, what counts as an answer and what will be ignored. A well-framed question can make a hard problem tractable; a poorly framed one can waste great effort on an answer no one needs. This article examines why questions matter so much, how good questions are formed, and how they are turned into the specific queries that surveys and other research methods put to the world.

The question comes first

The philosopher John Dewey argued that inquiry begins with a problematic situation, a sense that something is unsettled or unclear, and that the first task of inquiry is to turn that situation into a definite problem. He held that the way a problem is conceived decides which suggestions are entertained and which are dismissed, and that a problem well put is half solved (Dewey, 1938). The historian and philosopher R. G. Collingwood made a related point: every statement is an answer to a question, and a statement cannot be understood, or judged true or false, without knowing what question it was meant to answer (Collingwood, 1939).

Research on creativity supports this emphasis on framing. Getzels and Csikszentmihalyi (1976) studied art students and found that those who spent more time exploring and defining the problem before starting to work produced more original work, and were more successful years later, than those who settled quickly on a task. The ability to find and frame good problems, they concluded, is as important as the ability to solve them.

Firestein (2012), a neuroscientist, argued that science is driven less by what is known than by well-chosen ignorance. Good scientists, he suggested, are distinguished by the quality of the questions they ask about what is not yet known, not by the facts they have accumulated.

The error of the third kind

Statisticians speak of two kinds of error in testing a hypothesis: concluding that an effect exists when it does not, and missing one that does. Kimball (1957) proposed a third: giving the right answer to the wrong question. He observed that in statistical consulting, the most serious mistakes often arose not from faulty analysis but from misunderstanding what the client needed to know. Mitroff and Featheringham (1974) extended the idea to problem-solving in organisations, arguing that the most costly errors come from solving the wrong problem precisely.

Hand (1994) argued that many statistical questions are ambiguous until they are carefully taken apart. A question such as "Has the treatment improved outcomes?" may mean several different things, depending on which outcomes, for whom, compared with what and over what period. Different meanings lead to different analyses and can give different answers. Clarifying the question before collecting or analysing data, he argued, is one of the most valuable things a statistician can do.

The cost of poor questions is large. Chalmers and Glasziou (2009) estimated that a large share of investment in medical research is wasted, in part because research addresses questions that are not relevant to patients and clinicians or that have already been answered.

Diagram of five narrowing steps from a vague concern to an answerable question: concern, decision, research question, survey questions and analysis plan, each with an example about customer satisfaction with delivery.
Figure 1. From a vague concern to an answerable question. Based on Dewey (1938), Hand (1994) and Tourangeau et al. (2000); diagram by ModQu.

Forming a good research question

A useful research question has several properties. It is tied to a decision or purpose: someone will do something differently depending on the answer. It is specific: it names who or what is being studied, what is being measured and what comparison matters. It is answerable with evidence that can realistically be gathered. And it is open: the researcher does not already know the answer.

Medicine developed a simple structure for clinical questions, known by the initials PICO: the patient or population, the intervention, the comparison and the outcome (Richardson et al., 1995). The same structure helps in other fields. "Do customers prefer the new service?" becomes "Among customers who used both services in the past year, what proportion rate the new service as better than the old one for speed of delivery?"

Alvesson and Sandberg (2011) warned that many research questions simply fill small gaps in existing knowledge without challenging its assumptions. They argued that more interesting questions come from problematisation: identifying and questioning the assumptions that underlie a field or a way of thinking. In practical research, this might mean asking not only "How satisfied are customers with our delivery?" but "Is delivery what customers actually care about?"

From research questions to survey questions

A research question is rarely asked directly of the people being studied. It must be turned into specific questions that they can understand and answer accurately. Payne (1951), in an early guide to question wording, showed how small changes in words could change the answers people gave, and urged researchers to use simple, specific and neutral language. Schuman and Presser (1981) demonstrated in a series of experiments that the form, wording and order of questions could shift results substantially, for example depending on whether a "don't know" option was offered or whether questions about related topics came first.

Tourangeau et al. (2000) described the steps a person goes through in answering a survey question: understanding the question, retrieving relevant information from memory, forming a judgement and reporting an answer. Errors can arise at each step. A good survey question is designed so that each step is as easy and accurate as possible: its meaning is clear, it asks about things people can remember, and its answer options fit the way people think about the subject.

Questions to ask before asking questions

Before any study begins, a short set of questions helps to test whether the research question is ready:

Modern queries

The name ModQu stands for "Modern Queries". The phrase captures the idea that knowledge generation in the modern world depends on asking well-framed questions of the right people, in the right way, and analysing their answers honestly. Data are abundant; well-designed questions are not. Every survey, study and investigation is only as good as the questions behind it.

Conclusion

New knowledge begins with a question, and the quality of the question limits the quality of the answer. A good question is tied to a purpose, specific, answerable and open. Moving from a vague concern to a clear research question, and from that to precise survey questions and a plan for analysis, is the most important part of designing research. Getting the question right avoids the costliest error of all: a correct answer to the wrong question.

References

Alvesson, M., & Sandberg, J. (2011). Generating research questions through problematization. Academy of Management Review, 36(2), 247–271. https://doi.org/10.5465/amr.2011.59330882

Chalmers, I., & Glasziou, P. (2009). Avoidable waste in the production and reporting of research evidence. The Lancet, 374(9683), 86–89. https://doi.org/10.1016/S0140-6736(09)60329-9

Collingwood, R. G. (1939). An autobiography. Oxford University Press.

Dewey, J. (1938). Logic: The theory of inquiry. Henry Holt.

Firestein, S. (2012). Ignorance: How it drives science. Oxford University Press.

Getzels, J. W., & Csikszentmihalyi, M. (1976). The creative vision: A longitudinal study of problem finding in art. Wiley.

Hand, D. J. (1994). Deconstructing statistical questions. Journal of the Royal Statistical Society: Series A (Statistics in Society), 157(3), 317–356. https://doi.org/10.2307/2983526

Kimball, A. W. (1957). Errors of the third kind in statistical consulting. Journal of the American Statistical Association, 52(278), 133–142. https://doi.org/10.1080/01621459.1957.10501374

Mitroff, I. I., & Featheringham, T. R. (1974). On systemic problem solving and the error of the third kind. Behavioral Science, 19(6), 383–393. https://doi.org/10.1002/bs.3830190605

Payne, S. L. (1951). The art of asking questions. Princeton University Press.

Richardson, W. S., Wilson, M. C., Nishikawa, J., & Hayward, R. S. (1995). The well-built clinical question: A key to evidence-based decisions. ACP Journal Club, 123(3), A12–A13. https://doi.org/10.7326/ACPJC-1995-123-3-A12

Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. Academic Press.

Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press. https://doi.org/10.1017/CBO9780511819322