Skip to content

Intelligence · Analysis and tradecraft

Structured Analytic Techniques in Intelligence Analysis

Structured analytic techniques are methods for making an analyst's reasoning explicit, testable and open to challenge. This article explains where they came from, the main families of techniques, what the evidence says about whether they work and how to use them sensibly.

ModQu research article · 1,869 words · 20 references (APA 7th edition)

Download PDF

Intelligence analysis is the work of reaching judgements from incomplete, ambiguous and sometimes deceptive information. Most of that work happens inside analysts' heads, where it is fast and often good, but also vulnerable to predictable errors. Structured analytic techniques are methods designed to bring reasoning out of the head and onto the page: to list assumptions, lay out alternative explanations, test evidence against them and invite challenge. This article explains their origins, the main families of techniques, what research says about their effectiveness and how organisations can use them well.

Why structure analysis?

The case for structure rests on research into human judgement. Tversky and Kahneman (1974) showed that people rely on mental shortcuts, or heuristics, that usually serve them well but lead to systematic errors: judging likelihood by how easily examples come to mind, anchoring on an initial figure, and judging by resemblance while ignoring base rates. Nickerson (1998) reviewed extensive evidence of confirmation bias, the tendency to seek, interpret and remember information in ways that support existing beliefs.

Heuer (1999), drawing on this research for intelligence analysts, argued that such biases cannot be overcome simply by knowing about them. Analysts need tools that change the way they work: writing down assumptions, considering several hypotheses at once, and focusing on evidence that discriminates between hypotheses rather than evidence that is merely consistent with a favoured one. These tools became the core of what are now called structured analytic techniques.

Johnston (2005), in an ethnographic study of American intelligence analysts, found that formal methods were rarely used and that analysts relied mainly on their experience and on precedent. After intelligence failures in the early 2000s, reforms placed greater emphasis on analytic standards. The Office of the Director of National Intelligence (2015) analytic standards require, among other things, that products explain uncertainties, distinguish assumptions from judgements and incorporate analysis of alternatives, requirements that structured techniques help to meet.

The main families

A widely used primer produced for American intelligence analysts groups the techniques into three families (U.S. Government, 2009).

Table of three families of structured analytic techniques: diagnostic, contrarian and imaginative, with the purpose and examples of each.
Figure 1. Three families of structured analytic techniques. After U.S. Government (2009) and Pherson and Heuer (2021); table by ModQu.

Diagnostic techniques make reasoning transparent and test it. A key assumptions check lists the assumptions on which a judgement rests and asks how confident the analyst is in each and what would happen if it were wrong. A quality of information check reviews the reliability of the sources. Indicators are observable developments that would show which way a situation is moving. Analysis of competing hypotheses sets out several explanations and evaluates each piece of evidence against all of them.

Contrarian techniques challenge the current view. In devil's advocacy, someone builds the strongest case against the prevailing judgement. Team A/Team B sets two groups to argue different positions. High-impact/low-probability analysis examines how an unlikely but consequential event might come about. "What if?" analysis assumes an event has occurred and works backwards to see how it could have happened.

Imaginative techniques generate new possibilities. Structured brainstorming collects ideas before evaluating them. Outside-in thinking begins with the broad forces affecting an issue rather than the issue itself. Red team analysis tries to think as an adversary or competitor would. Alternative futures analysis develops several plausible scenarios.

Pherson and Heuer (2021) describe many more techniques, organised into families covering decomposition and visualisation, idea generation, scenarios and indicators, hypothesis generation and testing, assessment of cause and effect, challenge analysis, conflict management and decision support.

How techniques fit into analytic work

The techniques are not alternatives to one another. Different techniques support different stages of the same task.

Flow diagram of five stages of an analytic task, frame, generate, test, challenge and monitor, with suitable techniques for each stage.
Figure 2. Where techniques fit in an analytic task. Framework by ModQu, drawing on Pherson and Heuer (2021).

At the start, a key assumptions check and outside-in thinking help frame the problem. Brainstorming and hypothesis generation widen the range of possibilities. Analysis of competing hypotheses and quality of information checks test them. Devil's advocacy, a premortem, in which the team imagines that its judgement has turned out to be wrong and asks why (Klein, 2007), and red teaming challenge the result. Finally, indicators allow the judgement to be monitored and revised as events unfold. Dewar (2002), writing about planning rather than intelligence, makes a similar point: identifying load-bearing assumptions and the signposts that would show they are failing is one of the most practical ways to reduce avoidable surprises.

Do they work?

Despite their wide adoption, the evidence that structured techniques improve analytic accuracy is limited. The National Research Council (2011) concluded that intelligence agencies had adopted many analytic methods without testing them, and recommended that methods be evaluated using the standards of the behavioural sciences. Fischhoff and Chauvin (2011) assembled reviews of the relevant research, much of which had not been applied to intelligence settings.

Coulthart (2017) reviewed the empirical studies available for twelve core techniques and found that the evidence was thin and mixed. Some techniques had support from related research; others had little or none, and a few showed signs of making things worse in some conditions. Group brainstorming, for example, has long been found to produce fewer ideas than the same number of people working alone and pooling their results.

Experimental tests of analysis of competing hypotheses have been particularly sobering. Dhami et al. (2019) found that analysts trained in the method did not reach more accurate conclusions than those who were not, and that many did not follow its steps as intended. Mandel et al. (2018) found no accuracy benefit and some evidence that the method made probability judgements less coherent. Whitesmith (2020) found that it did not reliably reduce confirmation bias.

Chang et al. (2018) argue that many techniques were designed to reduce bias but neglect another major source of error: inconsistency, or noise, in judgements. They propose that techniques should be tested against measurable outcomes and redesigned where they fail.

The picture is not wholly negative. Schwenk (1990), in a meta-analysis of studies outside intelligence, found that devil's advocacy and dialectical inquiry, in which opposing positions are debated, generally led to better decisions than simply accepting expert recommendations. Tetlock and Gardner (2015) and Dhami et al. (2015) describe evidence from forecasting research that short training in probabilistic reasoning, combined with practice and feedback, can improve accuracy measurably.

Using techniques sensibly

The research suggests a balanced approach rather than either uncritical adoption or rejection:

  1. Choose techniques for a purpose. Ask what problem a technique solves: missing alternatives, untested assumptions, overconfidence, or failure to imagine change. Do not apply techniques as a ritual.
  2. Favour simple, well-supported practices. Writing down assumptions, considering alternatives, seeking disconfirming evidence and stating probabilities explicitly are supported by broad research and are cheap to do.
  3. Use challenge deliberately. Devil's advocacy and premortems work best when challenge is welcomed, and when the person challenging is protected from criticism for doing so.
  4. Record the reasoning. Even when a technique does not change a judgement, a written record makes the judgement easier for others to review and for the analyst to revisit.
  5. Measure results. Where possible, track the accuracy of judgements over time, so that the organisation learns which practices help.

Marrin (2012) argues that intelligence analysis would benefit from closer links between scholars, who can test methods, and practitioners, who know what problems need solving. Organisations outside government can apply the same principle by testing their own analytic practices on questions where outcomes can later be checked.

Common misuses

Structured techniques can be misused in ways that defeat their purpose:

A starter toolkit for small teams

Business intelligence teams rarely have time for elaborate methods. A small, practical toolkit covers most needs:

  1. Assumptions list. For each important judgement, list the three to five assumptions it depends on and mark which are weakest.
  2. Two or three hypotheses. Before settling on an explanation, write down at least two alternatives and identify the evidence that would distinguish between them.
  3. Probability and confidence. State how likely each outcome is, using numbers or a defined scale, and how confident the team is in the evidence.
  4. Premortem. For major judgements, spend twenty minutes imagining the judgement has proved wrong and listing why.
  5. Indicators. Name a handful of observable signs that would show the judgement needs revising, and assign someone to watch them.
  6. Review. Revisit judgements when the outcome is known, and record what was learned.

These six practices take little time, fit into ordinary reports and address the most common sources of analytic error: unexamined assumptions, premature closure on one explanation, vague uncertainty and failure to update.

An illustration

A retail bank's intelligence team is asked whether a rival is likely to launch a low-cost digital bank in the next year. The lead analyst's instinct is that it will, because the rival has hired several technology executives. Before writing, the team runs a key assumptions check and discovers that the judgement rests on the assumption that the hires are for a new venture rather than for upgrading existing systems. It then lists three hypotheses: a new digital bank, a rebuild of existing systems, and a partnership with a financial technology firm. Working through the evidence, it finds that a recent filing describes a large system modernisation programme, which fits the second hypothesis better than the first. A short premortem, imagining that no launch has occurred a year later, surfaces further reasons. The team reports a lower probability of launch than the analyst first expected, names the indicators that would change its view, such as an application for a new banking licence or a separate brand registration, and records its reasoning for review later. Twelve months on, the team checks the outcome against its judgement and notes which parts of its reasoning held up and which did not, adding the lesson to its guidance for future assessments.

Conclusion

Structured analytic techniques are tools for making reasoning visible and open to challenge. They grew from sound research on the limits of human judgement and are now part of standard analytic practice. But the evidence that individual techniques improve accuracy is weaker than their popularity suggests, and some may not work as intended. The most reliable gains come from simple disciplines, such as listing assumptions, considering alternatives, inviting challenge, stating probabilities and checking results, applied with judgement rather than as ritual.

References

Chang, W., Berdini, E., Mandel, D. R., & Tetlock, P. E. (2018). Restructuring structured analytic techniques in intelligence. Intelligence and National Security, 33(3), 337–356. https://doi.org/10.1080/02684527.2017.1400230

Coulthart, S. J. (2017). An evidence-based evaluation of 12 core structured analytic techniques. International Journal of Intelligence and CounterIntelligence, 30(2), 368–391. https://doi.org/10.1080/08850607.2016.1230706

Dewar, J. A. (2002). Assumption-based planning: A tool for reducing avoidable surprises. Cambridge University Press.

Dhami, M. K., Belton, I. K., & Mandel, D. R. (2019). The "analysis of competing hypotheses" in intelligence analysis. Applied Cognitive Psychology, 33(6), 1080–1090. https://doi.org/10.1002/acp.3550

Dhami, M. K., Mandel, D. R., Mellers, B. A., & Tetlock, P. E. (2015). Improving intelligence analysis with decision science. Perspectives on Psychological Science, 10(6), 753–757. https://doi.org/10.1177/1745691615598511

Fischhoff, B., & Chauvin, C. (Eds.). (2011). Intelligence analysis: Behavioral and social scientific foundations. National Academies Press. https://doi.org/10.17226/13062

Heuer, R. J., Jr. (1999). Psychology of intelligence analysis. Center for the Study of Intelligence, Central Intelligence Agency.

Johnston, R. (2005). Analytic culture in the US intelligence community: An ethnographic study. Center for the Study of Intelligence, Central Intelligence Agency.

Klein, G. (2007). Performing a project premortem. Harvard Business Review, 85(9), 18–19.

Mandel, D. R., Karvetski, C. W., & Dhami, M. K. (2018). Boosting intelligence analysts' judgment accuracy: What works, what fails? Judgment and Decision Making, 13(6), 607–621.

Marrin, S. (2012). Improving intelligence analysis: Bridging the gap between scholarship and practice. Routledge.

National Research Council. (2011). Intelligence analysis for tomorrow: Advances from the behavioral and social sciences. National Academies Press. https://doi.org/10.17226/13040

Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. https://doi.org/10.1037/1089-2680.2.2.175

Office of the Director of National Intelligence. (2015). Intelligence Community Directive 203: Analytic standards.

Pherson, R. H., & Heuer, R. J., Jr. (2021). Structured analytic techniques for intelligence analysis (3rd ed.). CQ Press.

Schwenk, C. R. (1990). Effects of devil's advocacy and dialectical inquiry on decision making: A meta-analysis. Organizational Behavior and Human Decision Processes, 47(1), 161–176. https://doi.org/10.1016/0749-5978(90)90051-A

Tetlock, P. E., & Gardner, D. (2015). Superforecasting: The art and science of prediction. Crown.

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131. https://doi.org/10.1126/science.185.4157.1124

U.S. Government. (2009). A tradecraft primer: Structured analytic techniques for improving intelligence analysis. Center for the Study of Intelligence.

Whitesmith, M. (2020). Cognitive bias in intelligence analysis: Testing the analysis of competing hypotheses method. Edinburgh University Press.