Skip to content

Intelligence · Analysis and tradecraft

Analysis of Competing Hypotheses

Analysis of competing hypotheses is a method for weighing several explanations against the evidence at the same time. This article explains how it works, why it focuses on disproving rather than proving, and what research says about its benefits and limits.

ModQu research article · 857 words · 8 references (APA 7th edition)

Download PDF

When analysts face an ambiguous situation, they tend to settle quickly on the most plausible explanation and then look for evidence that supports it. Analysis of competing hypotheses, usually shortened to ACH, is a method designed to counter that tendency. Developed by Richards Heuer and described in Heuer's book on the psychology of intelligence analysis (Heuer, 1999), it requires the analyst to consider several explanations at once and to judge each piece of evidence against all of them. This article explains the method, the reasoning behind it and the evidence on how well it works.

The problem it addresses

People test ideas by looking for confirming cases. Klayman and Ha (1987) showed that this "positive test strategy" is often reasonable, but can mislead when evidence that fits one hypothesis fits others equally well. Nickerson (1998) reviewed extensive research showing that people favour information consistent with what they already believe. In analysis, the result is that a judgement can accumulate a long list of "supporting" evidence, none of which actually distinguishes it from alternatives.

Heuer's insight was that the value of evidence lies in its diagnosticity: its ability to discriminate between hypotheses. Evidence consistent with every hypothesis tells us nothing about which is true.

The steps

Heuer (1999) set out ACH as eight steps:

  1. Identify the possible hypotheses, using a group if possible to widen the range.
  2. List the significant evidence and arguments, including assumptions and the absence of expected evidence.
  3. Build a matrix with hypotheses across the top and evidence down the side, and judge whether each item is consistent, inconsistent or neutral for each hypothesis.
  4. Refine the matrix, removing evidence that has no diagnostic value and reconsidering the hypotheses.
  5. Draw tentative conclusions by trying to disprove hypotheses, not prove them. The most likely hypothesis is usually the one with the least evidence against it.
  6. Analyse sensitivity: identify the few items of evidence on which the conclusion depends, and consider what would follow if they were wrong or deceptive.
  7. Report conclusions, including the relative likelihood of all hypotheses, not only the leading one.
  8. Identify milestones for future observation that would indicate events are taking a different course.
Matrix of four evidence items against three hypotheses for a competitor's price cut, marked consistent, inconsistent or neutral, with a row counting inconsistent items: zero for price war, two for clearing old stock and one for lower costs.
Figure 1. A small competing hypotheses matrix. The method counts evidence against each hypothesis, because evidence that fits everything does not help. Hypothetical example; method after Heuer (1999); table by ModQu.

What the evidence says

ACH became one of the most widely taught techniques in intelligence analysis (U.S. Government, 2009; Pherson & Heuer, 2021), but experiments have not shown clear benefits. Dhami et al. (2019) found that intelligence analysts trained in ACH did not reach more accurate judgements than untrained analysts, and that few followed all its steps. Mandel et al. (2018) found that ACH did not improve the accuracy of probability judgements and may have made them less internally consistent. Whitesmith (2020), in an experimental study, found that it did not reliably reduce confirmation bias.

Several explanations have been offered. Judging consistency for every cell is laborious and can itself be biased. Counting inconsistencies treats all evidence as equally important. And the method does not by itself produce calibrated probabilities.

A business example

Suppose a competitor suddenly cuts prices by 15 per cent. Three explanations are plausible: it has started a price war to win market share, it is clearing old stock before a new model, or it has lowered its costs and is passing the savings on. A manager's first instinct may be "price war", and plenty of evidence seems to fit: the cut is large, advertising has increased and sales staff report aggressive offers. But much of this evidence would also fit a stock clearance.

Laying the evidence out in a matrix changes the question from "what supports a price war?" to "what is inconsistent with each explanation?" The fact that the cut applies to every product line, including new ones, is hard to reconcile with a stock clearance. The absence of any new plant, supplier or process that could lower costs counts against the cost explanation. The price war hypothesis has the least evidence against it and so leads, but the analysis also shows which further evidence would be most useful: for example, whether the cut is temporary or permanent.

Using ACH well

These findings suggest using ACH as a thinking aid rather than a scoring machine. Its most valuable parts are the simplest: writing down more than one hypothesis, asking of each important item of evidence whether it actually discriminates between them, and focusing on the evidence that would most change the conclusion. Weighting critical items, checking them for deception and stating final judgements as probabilities address some of the method's weaknesses. ACH is most useful for questions with a small number of distinct explanations and contested evidence, and least useful when the evidence is thin or the hypotheses overlap heavily.

Conclusion

Analysis of competing hypotheses turns a sound principle into a procedure: consider several explanations and seek the evidence that tells them apart. The procedure's benefits are not yet supported by experimental evidence, but the principle remains one of the most useful habits an analyst can adopt.

References

Dhami, M. K., Belton, I. K., & Mandel, D. R. (2019). The "analysis of competing hypotheses" in intelligence analysis. Applied Cognitive Psychology, 33(6), 1080–1090. https://doi.org/10.1002/acp.3550

Heuer, R. J., Jr. (1999). Psychology of intelligence analysis. Center for the Study of Intelligence, Central Intelligence Agency.

Klayman, J., & Ha, Y.-W. (1987). Confirmation, disconfirmation, and information in hypothesis testing. Psychological Review, 94(2), 211–228. https://doi.org/10.1037/0033-295X.94.2.211

Mandel, D. R., Karvetski, C. W., & Dhami, M. K. (2018). Boosting intelligence analysts' judgment accuracy: What works, what fails? Judgment and Decision Making, 13(6), 607–621.

Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. https://doi.org/10.1037/1089-2680.2.2.175

Pherson, R. H., & Heuer, R. J., Jr. (2021). Structured analytic techniques for intelligence analysis (3rd ed.). CQ Press.

U.S. Government. (2009). A tradecraft primer: Structured analytic techniques for improving intelligence analysis. Center for the Study of Intelligence.

Whitesmith, M. (2020). Cognitive bias in intelligence analysis: Testing the analysis of competing hypotheses method. Edinburgh University Press.