Skip to content

Intelligence · Analysis and tradecraft

Detecting Information Gaps Before Making a Strategic Judgment

Strategic judgements often go wrong not because of what analysts know but because of what they fail to notice they do not know. This article explains why missing information is so easily overlooked, how to find gaps systematically, how to decide which ones are worth closing and how to report the gaps that remain.

ModQu research article · 2,543 words · 23 references (APA 7th edition)

Download PDF

Every strategic judgement is made with incomplete information. That is unavoidable. The danger lies elsewhere: in failing to notice what is missing. An analyst who knows that a key fact is unknown can say so, estimate around it and look for it. An analyst who does not notice the gap will reason confidently from whatever evidence happens to be available, and may reach a firm conclusion on a foundation with a hole in the middle.

This article explains why people overlook missing information, describes methods for detecting gaps before a judgement is made, shows how to decide which gaps are worth closing and suggests how to report the gaps that cannot be closed.

Why gaps go unnoticed

Out of sight, out of mind

Fischhoff et al. (1978) conducted a classic set of experiments using fault trees, diagrams that list the possible causes of a problem such as a car failing to start. When some branches of the tree were removed and replaced with a general category of "all other problems", participants, including experienced mechanics, assigned far too little probability to that category. What was not shown was largely ignored. The researchers described this as an "out of sight, out of mind" effect, and it applies directly to intelligence: the possibilities and facts not in front of the analyst receive too little weight.

Judging from what is available

Tversky and Kahneman (1974) showed that people estimate the frequency or likelihood of events by how easily examples come to mind. Information that is vivid, recent or simply present is weighted heavily; information that is absent cannot be weighed at all. Sanbonmatsu et al. (1992) studied how people evaluate products when some attributes are missing. They found that people often failed to take account of the missing attributes and formed confident, extreme judgements from the information they had. Those with more knowledge of the product category were more likely to notice what was missing, because they knew what information should have been there.

Looking for confirmation

Nickerson (1998), reviewing research on confirmation bias, showed that people tend to seek information that would confirm their current hypothesis rather than information that could disprove it. This creates systematic gaps: the evidence that would challenge a view is the evidence least likely to be sought. Heuer (1999) observed that analysts also rarely ask what evidence is absent that should be present if their hypothesis were true.

Organisational gaps

Some gaps exist only from the analyst's point of view. The information exists somewhere in the organisation but never reaches the person making the judgement. The 9/11 Commission found that American agencies held important pieces of information about the attackers that were never brought together (National Commission on Terrorist Attacks Upon the United States, 2004). In business, the sales team may know what customers say about a rival, while the strategy team, writing an assessment of that rival, never asks.

Kinds of not knowing

It helps to distinguish different kinds of ignorance. Smithson (1989) developed a detailed classification, separating, for example, incompleteness of information from distortion and from matters judged irrelevant. A simpler framework, widely used in planning, distinguishes four categories according to whether the organisation holds the information and whether it is aware of the question.

Two by two matrix crossing whether the organisation holds the information with whether it is aware of the question, giving known knowns, known unknowns, unknown knowns and unknown unknowns.
Figure 1. Four kinds of knowledge and ignorance. Gap detection tries to turn unknown unknowns into known unknowns, and to bring unknown knowns to the analyst's attention. Framework by ModQu, drawing on Smithson (1989) and Feduzi and Runde (2014).

Known unknowns are gaps the analyst has identified. They can be listed, assessed and, perhaps, closed. Unknown unknowns are gaps no one has noticed. Feduzi and Runde (2014) argue that these are not simply beyond reach: structured methods of questioning can bring many of them to light, converting them into known unknowns. Unknown knowns are facts the organisation holds but the analyst does not see, which can be found by better sharing and search.

These categories connect to an older distinction in economics. Knight (1921) separated risk, in which probabilities can be estimated, from uncertainty, in which they cannot. Ellsberg (1961) showed that people dislike ambiguity, situations where probabilities themselves are unknown, and will often pay to avoid it. Gaps in information create ambiguity, and recognising them openly is the first step towards managing it. Taleb (2007) argued that the events with the greatest impact often come from outside the range of possibilities that anyone considered, which is a reason for humility about how complete any map of gaps can be.

Puzzles, mysteries and gaps

Treverton (2003) distinguished puzzles, which have a definite answer that more information could reveal, from mysteries, which depend on future choices and have no answer yet. The distinction matters for gap detection. A gap in a puzzle, such as how many units a competitor sold last year, can in principle be closed by collection. A gap in a mystery, such as how the competitor will respond to a price cut, cannot be closed by collection alone, because the answer does not yet exist. It can only be narrowed by understanding the actor better and by preparing for several possibilities.

Methods for detecting gaps

Start from a model

Clark (2020) argues that analysis should be built around an explicit target model: a representation of the person, organisation or system being studied, showing its parts, relationships and processes. A good model makes gaps visible, because it shows which parts are well understood and which are blank. If the model of a competitor shows its products, customers, costs and leadership, but the analyst has no information about its costs, the gap is obvious in a way it would not be in a narrative report.

Ask what should be there

Heuer (1999) recommended that analysts ask, for each hypothesis, what evidence they would expect to see if it were true, and then check whether that evidence is present. Absent evidence can be highly diagnostic. Grabo (2002) noted that in warning analysis, the absence of preparations that an attack would require is an important indicator. But Grabo also warned that the inability to find evidence is not the same as evidence of absence, especially when the adversary is concealing its activities.

The Iraq case shows the danger of getting this wrong in the other direction. American analysts treated the lack of evidence of Iraqi weapons not as a sign that they might not exist, but as evidence of successful concealment (Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction, 2005). Absent evidence must be interpreted carefully, considering both the possibility that it reflects reality and the possibility that collection has failed.

Structured questioning

Several structured techniques help surface unknown unknowns. Pherson and Heuer (2021) describe starbursting, in which a team generates questions beginning with who, what, where, when, why and how about the issue, before trying to answer any of them. The key assumptions check asks what must be true for the judgement to hold, revealing premises for which there is no evidence. The premortem asks a team to imagine that its judgement has turned out to be wrong and to explain why (Klein, 2007); the explanations often point to gaps no one had noticed. In assumption-based planning, Dewar (2002) recommends identifying the load-bearing assumptions of a plan and asking which ones are vulnerable, a process that frequently reveals missing information.

Feduzi and Runde (2014) propose an approach inspired by the philosopher Francis Bacon: rather than seeking evidence that supports a favoured hypothesis, decision-makers should actively generate alternative hypotheses and seek evidence that would eliminate them. Each alternative implies different questions, and the questions reveal what is not known.

Check the evidence chain

Schum (1994) emphasised that every inference rests on a chain of evidence, from the original event through sources and reports to the analyst's conclusion. Gaps can appear at any link. Did anyone actually observe the event? Was the source in a position to know? Has the report been accurately translated and summarised? Mapping the chain for each key judgement reveals where it depends on assumptions rather than evidence.

Ask outside the team

Finally, analysts should ask colleagues in other parts of the organisation, and outside experts, what they know that bears on the question. This is the simplest way to find unknown knowns. Lowenthal (2022) notes that the formal statement of intelligence requirements exists partly to ensure that collectors know what analysts need; the same principle applies informally within any organisation.

Organising a gap review

A gap review need not be elaborate. For most strategic questions, a single meeting of one to two hours, held once a draft judgement exists but before it is final, is enough. A practical agenda has four parts.

First, the lead analyst states the draft judgement and the main evidence for it in a few minutes. Second, the group works through the model of the target or market, marking each element as well supported, weakly supported or blank. Third, the group runs one structured exercise, such as starbursting or a premortem, to surface questions that the model did not reveal. Fourth, each gap is entered in the register with an owner and an action.

It helps to invite at least one person who did not work on the analysis, since those closest to a judgement are least likely to see its gaps. Recording the review also creates a record that can be checked later: if a judgement proves wrong, the team can see whether the cause was a gap it identified and accepted, or one it never noticed.

Gaps in survey and market research

Information gaps are especially common in business research because so much of it relies on data that happen to exist, such as sales records, web analytics and published reports. These sources describe current customers well but say little about people who do not buy, about why customers leave, or about how they would react to a product that does not yet exist. Well-designed surveys can close many of these gaps directly, by asking the people whose views are missing. Equally important, survey design itself benefits from gap analysis: listing what the decision needs to know before writing the questionnaire prevents the common result of a survey that answers interesting questions but not the ones that matter.

Deciding which gaps to close

Not every gap is worth closing. Collection costs time and money, and decisions cannot always wait. Decision analysis offers a principled way to choose. Howard (1966) set out information value theory: the value of a piece of information is the expected improvement it would bring to the decision, compared with deciding without it. Raiffa (1968) showed how to calculate the value of perfect information, which sets an upper limit on what any research is worth.

In practice, a full calculation is rarely needed. The key insight is that information is valuable only if it could change the decision. A gap that is large but would not alter the choice is not worth much effort. A small gap that could tip the balance is worth a great deal. Analysts can therefore assess each gap by two simple questions: how much could closing it change the judgement, and how feasible and costly is it to close?

Table of five information gaps for a market entry decision, showing for each whether it could change the decision, whether it can be closed and the planned action, from a customer survey to leaving the gap open.
Figure 2. A simple information gap register. Gaps that could change the decision and can be closed quickly are dealt with first; gaps that cannot be closed are reported as key uncertainties. Hypothetical example by ModQu.

A gap register, a short table listing each gap, its potential effect, whether it can be closed and the planned action, turns this reasoning into a practical tool. It also provides a record that can be shared with decision-makers.

Reporting what remains unknown

Some gaps will remain when the judgement must be made. These should be reported, not hidden. Good practice includes:

  1. State the key gaps explicitly. A short section in an assessment titled "What we do not know" tells readers where the judgement is weakest.
  2. Explain how the gaps affect confidence. A judgement may be likely on the available evidence, but held with only moderate confidence because important information is missing.
  3. Say what would change the judgement. Identify the information that, if obtained, would most alter the conclusion, and the indicators that would reveal it.
  4. Distinguish puzzles from mysteries. Make clear which gaps could be closed with more collection and which depend on future decisions by others.

Betts (2007) argues that no amount of reform can eliminate uncertainty from strategic intelligence. Honest reporting of gaps does not remove uncertainty, but it allows decision-makers to see it and to plan accordingly.

Common mistakes

An illustration

A Canadian manufacturer of outdoor equipment is considering entering the Australian market. The strategy team drafts an assessment recommending entry, based on market size data, favourable exchange rates and strong online reviews of similar products. Before the board meeting, the team carries out a gap review.

Using a simple model of the market, they find that they have good information about demand but almost none about distribution: they do not know whether the main retail chains would stock their products or on what terms. A premortem exercise surfaces another gap: no one has considered how the leading Australian brand might respond. A conversation with the company's own sales staff reveals an unknown known: a former employee now works for an Australian distributor and could provide insight.

The team builds a gap register. The price that Australian customers would accept could change the decision and can be measured quickly, so it commissions a survey. Distribution terms can be explored through interviews, starting with the former employee's contacts. The competitor's response cannot be known in advance, so the team develops scenarios and indicators to watch. A more precise estimate of total market size would be costly and would not change the decision, so the team accepts the existing range. The final assessment recommends a phased entry, states the remaining gaps clearly and sets out what would cause the company to accelerate or pull back.

Conclusion

Missing information is easy to overlook because the mind works with what is in front of it. Research on fault trees, missing attributes and confirmation bias shows how consistently people neglect what they cannot see. Systematic methods help: building explicit models, asking what evidence should be present, using structured questioning, mapping evidence chains and asking others what they know. Not every gap needs to be closed; the ones that matter are those that could change the decision. And the gaps that remain should be stated plainly, so that decision-makers can judge how much weight to put on the conclusion and what to watch for as the situation develops.

References

Betts, R. K. (2007). Enemies of intelligence: Knowledge and power in American national security. Columbia University Press.

Clark, R. M. (2020). Intelligence analysis: A target-centric approach (6th ed.). CQ Press.

Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction. (2005). Report to the President of the United States. U.S. Government Printing Office.

Dewar, J. A. (2002). Assumption-based planning: A tool for reducing avoidable surprises. Cambridge University Press.

Ellsberg, D. (1961). Risk, ambiguity, and the Savage axioms. Quarterly Journal of Economics, 75(4), 643–669. https://doi.org/10.2307/1884324

Feduzi, A., & Runde, J. (2014). Uncovering unknown unknowns: Towards a Baconian approach to management decision-making. Organizational Behavior and Human Decision Processes, 124(2), 268–283. https://doi.org/10.1016/j.obhdp.2014.04.001

Fischhoff, B., Slovic, P., & Lichtenstein, S. (1978). Fault trees: Sensitivity of estimated failure probabilities to problem representation. Journal of Experimental Psychology: Human Perception and Performance, 4(2), 330–344. https://doi.org/10.1037/0096-1523.4.2.330

Grabo, C. M. (2002). Anticipating surprise: Analysis for strategic warning. Joint Military Intelligence College.

Heuer, R. J., Jr. (1999). Psychology of intelligence analysis. Center for the Study of Intelligence, Central Intelligence Agency.

Howard, R. A. (1966). Information value theory. IEEE Transactions on Systems Science and Cybernetics, 2(1), 22–26. https://doi.org/10.1109/TSSC.1966.300074

Klein, G. (2007). Performing a project premortem. Harvard Business Review, 85(9), 18–19.

Knight, F. H. (1921). Risk, uncertainty and profit. Houghton Mifflin.

Lowenthal, M. M. (2022). Intelligence: From secrets to policy (9th ed.). CQ Press.

National Commission on Terrorist Attacks Upon the United States. (2004). The 9/11 Commission report. W. W. Norton.

Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. https://doi.org/10.1037/1089-2680.2.2.175

Pherson, R. H., & Heuer, R. J., Jr. (2021). Structured analytic techniques for intelligence analysis (3rd ed.). CQ Press.

Raiffa, H. (1968). Decision analysis: Introductory lectures on choices under uncertainty. Addison-Wesley.

Sanbonmatsu, D. M., Kardes, F. R., & Herr, P. M. (1992). The role of prior knowledge and missing information in multiattribute evaluation. Organizational Behavior and Human Decision Processes, 51(1), 76–91. https://doi.org/10.1016/0749-5978(92)90005-R

Schum, D. A. (1994). The evidential foundations of probabilistic reasoning. Wiley.

Smithson, M. (1989). Ignorance and uncertainty: Emerging paradigms. Springer.

Taleb, N. N. (2007). The black swan: The impact of the highly improbable. Random House.

Treverton, G. F. (2003). Reshaping national intelligence for an age of information. Cambridge University Press.

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131. https://doi.org/10.1126/science.185.4157.1124