Skip to content

Insights · Insight quality and pitfalls

Confirmation Bias and the Search for Expected Insights

How the tendency to seek and favour evidence that supports existing beliefs enters every stage of insight work, from framing questions to reporting results, and practical safeguards that help analysts find what is true rather than what is expected.

ModQu research article · 915 words · 10 references (APA 7th edition)

Download PDF

Insight work is supposed to reveal something new. Yet analysts, like everyone else, tend to find what they expect. Confirmation bias, the tendency to search for, interpret and remember information in ways that support existing beliefs, is one of the best documented findings in psychology. Nickerson (1998) reviewed the evidence and described it as a pervasive phenomenon that appears in many guises, from everyday reasoning to science, medicine and law. In insight work it is especially dangerous, because the people commissioning research often have a view they hope to see confirmed.

How confirmation bias works

Wason (1960) gave participants a sequence of three numbers, 2, 4, 6, and asked them to discover the rule that generated it by proposing other sequences and being told whether they fitted. Most participants formed a hypothesis, such as "numbers increasing by two", and tested it only with sequences that fitted their hypothesis. Because the real rule was simply "any increasing numbers", every test was confirmed, and many participants announced the wrong rule with confidence. Klayman and Ha (1987) explained that this "positive test strategy" is often reasonable, but fails when the hypothesis is narrower than the truth, because only tests that could disconfirm it reveal the error.

People also evaluate evidence unevenly. Lord et al. (1979) showed participants mixed evidence about the death penalty and found that both supporters and opponents rated studies supporting their own view as more convincing, and became more polarised after reading the same evidence. Kunda (1990) reviewed research on motivated reasoning and showed that when people want a particular conclusion, they apply stricter standards to evidence against it. Hart et al. (2009) conducted a meta-analysis of studies on selective exposure and found that people generally prefer information that supports their existing attitudes, especially when those attitudes are strongly held.

Where it enters insight work

Table of five stages of insight work with how confirmation bias enters each and a safeguard: framing the question (asking only whether the expected answer is true; frame competing hypotheses), choosing data (selecting sources likely to agree; decide sources in advance), analysis (trying many cuts until one fits; pre-specify the main analyses), interpretation (explaining away awkward results; ask what result would have changed your mind), and reporting (quoting only supportive evidence; report contrary findings).
Figure 1. Confirmation bias can enter at every stage of insight work. A safeguard at each stage reduces its influence. Diagram by ModQu.

Bias in analysis is particularly hard to see. Simmons et al. (2011) showed that common flexible practices, such as trying several outcome measures, adding observations until results become significant, or dropping conditions, can make it easy to find a "significant" result for almost any hypothesis. An analyst who explores a data set looking for support for an expected insight will usually find some.

An example

Consider a hypothetical retailer whose marketing director believes that a new loyalty app has increased customer spending. The insight team is asked to "show the impact of the app". The framing already assumes the answer. The team compares app users with non-users and finds that users spend 30% more. It also finds a few quotes from customers praising the app, which go into the report.

A colleague reviewing the analysis asks what result would have shown that the app had no effect. The answer is that customers who download a loyalty app were probably the retailer's most engaged customers already. When the team compares the same customers' spending before and after they downloaded the app, and against similar customers who did not, the increase shrinks to around 3%. Survey comments include as many complaints about the app as praise. The final report presents both analyses and explains why the first overstated the effect. The app is kept, but its budget is set on the basis of the smaller, more credible estimate.

The example shows how bias can enter without anyone intending to mislead. The question was framed around an expected answer, the first comparison happened to support it, and supportive quotes were easy to find. Only an explicit attempt to find disconfirming evidence revealed the problem.

Safeguards

Several practices reduce the influence of confirmation bias.

State competing hypotheses. Heuer (1999) recommended analysing competing hypotheses explicitly, listing all plausible explanations and focusing on evidence that discriminates between them, rather than evidence consistent with the favoured one.

Consider the opposite. Lord et al. (1984) found that asking people to consider how they would have judged the evidence if it had pointed the other way reduced biased evaluation.

Decide in advance. Writing down the main questions, data sources and analyses before looking at the results, an approach known in science as preregistration, separates confirmatory analysis from exploration. Nosek et al. (2018) argued that preregistration makes clear which findings were predicted and which were discovered after the fact.

Ask what would change your mind. Before analysis, identify the result that would lead to a different conclusion. If no possible result would do so, the analysis is not a test.

Invite challenge. Ask a colleague who did not frame the question to review the analysis and argue for an alternative interpretation.

Separate the analyst from the sponsor. Where possible, the person who wants a particular answer should not be the person who designs and runs the analysis. When this is not possible, agreeing the method with the sponsor before results are known protects both sides: the sponsor cannot later reject an unwelcome result on grounds of method, and the analyst is not tempted to adjust the method to please the sponsor.

Key takeaways

Confirmation bias leads analysts to seek, interpret and report evidence in ways that support what they already believe. It can enter at every stage of insight work, from framing questions to reporting. Explicitly stating competing hypotheses, considering the opposite, deciding analyses in advance, identifying what would change one's mind and inviting challenge all help analysts find what is true rather than what is expected.

References

Hart, W., Albarracín, D., Eagly, A. H., Brechan, I., Lindberg, M. J., & Merrill, L. (2009). Feeling validated versus being correct: A meta-analysis of selective exposure to information. Psychological Bulletin, 135(4), 555–588. https://doi.org/10.1037/a0015701

Heuer, R. J., Jr. (1999). Psychology of intelligence analysis. Center for the Study of Intelligence, Central Intelligence Agency.

Klayman, J., & Ha, Y.-W. (1987). Confirmation, disconfirmation, and information in hypothesis testing. Psychological Review, 94(2), 211–228. https://doi.org/10.1037/0033-295X.94.2.211

Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108(3), 480–498. https://doi.org/10.1037/0033-2909.108.3.480

Lord, C. G., Lepper, M. R., & Preston, E. (1984). Considering the opposite: A corrective strategy for social judgment. Journal of Personality and Social Psychology, 47(6), 1231–1243. https://doi.org/10.1037/0022-3514.47.6.1231

Lord, C. G., Ross, L., & Lepper, M. R. (1979). Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology, 37(11), 2098–2109. https://doi.org/10.1037/0022-3514.37.11.2098

Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. https://doi.org/10.1037/1089-2680.2.2.175

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632

Wason, P. C. (1960). On the failure to eliminate hypotheses in a conceptual task. Quarterly Journal of Experimental Psychology, 12(3), 129–140. https://doi.org/10.1080/17470216008416717