The Generation of Knowledge · Knowledge and decisions
Knowledge Gaps: How to Identify and Prioritize What Is Unknown
What knowledge gaps are, the different kinds of ignorance, systematic ways to find gaps through reviews and evidence maps, why research effort is often wasted on the wrong questions, and practical methods for deciding which gaps to fill first.
Every decision rests on a mixture of what is known and what is not. Organisations, researchers and governments constantly face the question of where to direct their efforts to learn more. Some gaps in knowledge matter enormously; others matter little. Some can be closed with modest effort; others resist even large investments. Choosing well among them is one of the most consequential decisions in research, yet it is often made informally, by habit or by whoever is most persuasive. This article examines what knowledge gaps are, how they can be identified systematically and how they can be prioritised so that effort goes where it will make the greatest difference.
Kinds of not knowing
Not all ignorance is the same. Smithson (1989) developed a taxonomy of ignorance, distinguishing, for example, between error, in which what is believed is wrong, and incompleteness, in which something is missing. Within incompleteness he distinguished uncertainty, where the possibilities are known but not which will occur, from absence, where relevant matters are not even recognised. Gross (2010) studied how ignorance and surprise shape research and decision-making, emphasising that recognising what we do not know is itself a form of knowledge.
Merton (1987) introduced the term specified ignorance: the express recognition of what is not yet known but needs to be known in order to lay the foundation for still more knowledge. He argued that specifying ignorance is an essential step in science, because a well-defined question directs research in a way that vague curiosity cannot. Firestein (2012) went further, arguing that science is driven more by ignorance than by knowledge: researchers are motivated by well-formulated questions about what is not known, and good science often produces more new questions than answers.
Some ignorance is not accidental. Proctor and Schiebinger (2008) introduced the study of agnotology, the deliberate or structural production of ignorance. They documented cases in which uncertainty was manufactured, for example by industries that funded research designed to cast doubt on evidence of harm, and cases in which ignorance persisted because certain questions were never asked or funded.
A useful practical distinction is between three kinds of gap:
Known gaps, questions that have been recognised but not yet answered.
Unrecognised gaps, questions that no one has thought to ask, often because existing frameworks or categories hide them.
False knowledge, beliefs that are widely held but wrong, which create the illusion that no gap exists.
The second and third kinds are the most dangerous, because they do not prompt any search for evidence. Kahneman (2011) described a tendency he summarised as "what you see is all there is": people form confident judgements from the information available to them without considering what information might be missing.
Finding gaps systematically
Systematic reviews
The most rigorous way to find gaps in research is to review what is already known. A systematic review uses explicit, reproducible methods to find, select and appraise all relevant studies on a question (Higgins et al., 2019; Petticrew & Roberts, 2006). In doing so, it shows not only what the evidence says but also where evidence is missing, weak or inconsistent.
Robinson et al. (2011) developed a framework for identifying research gaps from systematic reviews. They proposed that each gap be characterised by the elements of the research question it concerns, such as the population, the intervention, the comparison and the outcomes, and by the reason it exists. They distinguished four reasons: insufficient or imprecise information, where there are too few studies or too few participants; biased information, where the available studies have serious flaws; inconsistent results, where studies disagree; and not the right information, where studies exist but do not address the populations, outcomes or comparisons that matter for decisions.
Evidence maps
Where a full systematic review is too costly, an evidence map can show the distribution of research across a field. Miake-Lye et al. (2016) reviewed published evidence maps and found that they typically present the volume and characteristics of research in a visual form, such as a grid showing which combinations of topics and outcomes have been studied and which have not. Snilstveit et al. (2016) described evidence and gap maps developed for international development, which plot interventions against outcomes and show at a glance where evidence is plentiful, where it is thin and where it is absent. Such maps help funders and researchers see empty cells that deserve attention.
Beyond gap-spotting
Sandberg and Alvesson (2011) studied how researchers in management formulated research questions and found that most did so by gap-spotting: identifying something that previous studies had not addressed and filling it. They argued that this approach tends to produce incremental contributions that extend existing theory without questioning it. Alvesson and Sandberg (2011) proposed an alternative, problematisation, in which researchers identify and challenge the assumptions underlying existing research. This approach is more likely to reveal unrecognised gaps and false knowledge, because it asks not "what has not been studied?" but "what has been taken for granted?".
Looking for weak signals
In organisations, gaps often appear first as faint signs that something is changing. Ansoff (1975) argued that firms could avoid strategic surprises by paying attention to weak signals, early and imprecise indications of change, and responding in graduated steps as the signals grew stronger. Day and Schoemaker (2006) described how organisations can develop peripheral vision, scanning the edges of their environment for developments that could become important.
Asking those affected
Some gaps are visible only to the people who experience a problem. In health research, the James Lind Alliance brings together patients, carers and clinicians to identify and rank the unanswered questions they consider most important (Cowan & Oliver, 2021). Such exercises have often identified priorities that differed from those of researchers, for example placing more emphasis on everyday management of a condition and less on new drugs.
Research methods themselves can uncover unrecognised gaps. Open-ended survey questions, in-depth interviews and focus groups allow people to raise issues that researchers had not anticipated. A question such as "What else should we have asked you about?" at the end of a survey often reveals topics that a structured questionnaire missed. Qualitative work before a large survey is valuable for this reason: it helps ensure that the survey asks about the things that matter to respondents, not only those that occurred to its designers.
Gaps that are not really gaps
Some apparent gaps exist only because knowledge cannot be found. Huber (1991) observed that organisations often do not know what they know: information gathered by one unit is unknown to another, studies are filed and forgotten, and expertise held by individuals is invisible to colleagues. Before commissioning new research, it is worth searching the organisation's own archives, asking colleagues who have worked on related problems and checking whether partners or industry bodies have already studied the question. Closing such gaps costs far less than collecting new data.
Why gaps are often left unfilled
Research waste
Chalmers and Glasziou (2009) estimated that a large proportion of investment in medical research was wasted, through four main routes: asking questions that were not relevant to patients and clinicians, using inappropriate designs and methods, failing to publish results, and reporting results in ways that were biased or unusable. They suggested that the cumulative effect of these losses could amount to as much as 85% of the investment. A central part of the problem was that new studies were often started without first reviewing what was already known, so that some questions were studied repeatedly while others were neglected.
Chalmers et al. (2014) set out recommendations for increasing the value of research by setting priorities better. These included basing new research on systematic assessments of existing evidence, involving the users of research in setting priorities and making the process transparent.
Unreliable findings
Some apparent knowledge is not reliable. Ioannidis (2005) argued that in many fields, a large share of published research findings are likely to be false, because of small studies, small effects, flexible methods and the preference of journals for positive results. The Open Science Collaboration (2015) attempted to reproduce 100 studies published in leading psychology journals and found that fewer than half of the replications produced statistically significant results, and that effects were on average about half as large as originally reported. Findings that have not been replicated may therefore conceal gaps behind an appearance of knowledge.
Incentives
Researchers and organisations are rewarded for some kinds of knowledge more than others. Novel findings are more publishable than replications; questions that interest funders are more likely to be studied than those that matter to people without resources. These incentives can leave important gaps unfilled for long periods.
Prioritising gaps
Once gaps are identified, they must be ranked. Resources are limited, and not every gap is worth filling.
Criteria
Viergever et al. (2010) reviewed approaches to setting priorities for health research and identified common elements of good practice. These included being clear about the context and purpose, involving appropriate stakeholders, gathering information on existing evidence, using explicit criteria, and being transparent about how decisions were made. The criteria used in many priority-setting exercises fall into a few groups:
Importance. How much does the gap matter to decisions, outcomes or people's lives?
Uncertainty. How little is known, and how much could research reduce the uncertainty?
Feasibility. Can the gap be closed with available methods, data and resources, and in time to be useful?
Cost. What would it cost to close the gap, compared with the value of doing so?
Equity. Does the gap particularly affect groups whose needs are often neglected?
Figure 1 combines the first two criteria into a simple matrix.
The value of information
Decision analysis offers a more formal approach. The expected value of information measures how much a decision would improve if a particular uncertainty were reduced (Claxton, 1999). By calculating this value for each uncertain quantity in a decision, analysts can identify which gaps matter most. Claxton and Sculpher (2006) showed how this method could be used to prioritise health research, by comparing the expected value of research on different questions with its cost. Keisler et al. (2014) reviewed applications in environmental and other fields and found the approach useful, though demanding to apply.
Hubbard (2014) applied similar reasoning in business and argued that organisations should first identify the uncertainties with the highest information value, which are often not the ones they currently measure. A short calculation, even with rough estimates, frequently reveals that one or two uncertainties dominate a decision and deserve almost all of the measurement effort.
Structured judgement
Where formal analysis is not possible, structured methods for combining expert and stakeholder judgement can help. Groups can rate gaps against explicit criteria, combine their ratings and discuss differences before reaching a final ranking. Such methods reduce the influence of the most persuasive individual and make the basis of decisions visible.
Testing assumptions
Prioritisation can itself be biased. Klein (2007) proposed the premortem: imagining that a decision has failed and asking why. Applied to research priorities, a premortem might ask: "Suppose that in three years we discover we studied the wrong things. What did we overlook?" This exercise encourages people to look for unrecognised gaps and assumptions. Tetlock and Gardner (2015) found that the most accurate forecasters actively sought information that might prove them wrong, a habit that also helps in identifying what is unknown.
Keeping track of gaps over time
Knowledge gaps are not fixed. New research closes some, new developments open others, and gaps that once seemed unimportant can become urgent when circumstances change. Elliott et al. (2014) proposed living systematic reviews, which are updated continually as new evidence appears, so that the picture of what is known and unknown stays current. Organisations can apply the same idea on a smaller scale by maintaining a register of open questions, reviewing it at regular intervals and noting when a gap has been closed, when it has grown more important and when new gaps have appeared.
Common mistakes
Experience suggests several common mistakes in dealing with knowledge gaps.
Measuring what is easy. Effort is directed at questions that are simple to answer rather than at those that matter most.
Confusing interest with importance. A question can be fascinating without being consequential for any decision.
Ignoring what is already known. New studies are started without a review of existing evidence, so that effort is duplicated.
Treating absence of evidence as evidence of absence. The fact that no study has found an effect does not mean that no effect exists, particularly if few or small studies have looked.
Overlooking the people affected. Priorities set only by researchers or managers can miss gaps that are obvious to those who experience the problem.
A worked example
Consider a professional association that has seen membership renewals fall for three years. Its board wants to know why and what to do.
The first step is to set out what is already known. The association's records show which members have left, their length of membership, region and type of employment. An earlier member survey asked about satisfaction with services. Published research on membership organisations suggests common reasons for leaving, such as cost, changes in employment and the availability of free alternatives.
The second step is to specify the gaps. The records show who left but not why. The earlier survey measured satisfaction among those who stayed, not those who left. And no one has examined whether the decline is concentrated in particular groups, such as younger members or those in certain regions.
The third step is to rank the gaps. The board must decide whether to change its fees, its services or its communications. The reason members leave matters most to that decision and is the least known, so it is the research priority. Whether the decline is concentrated in certain groups can be answered quickly from existing records at almost no cost, so it should be done first. The general findings from published research are already well established and can inform the design of the new study without further work.
The result is a focused plan: an analysis of existing records, followed by a short survey of recent former members and a sample of current members, designed to compare their reasons and to test the specific options the board is considering. The research is shaped by the gaps that matter, rather than by a general wish to know more.
Knowledge gaps in survey and market research
For organisations that commission or conduct surveys, identifying and prioritising gaps is the starting point of good research.
Start with what is known. Before designing a new study, review existing evidence: previous surveys, official statistics, published research and the organisation's own data. Many questions can be answered, at least in part, from existing sources, and the review shows exactly where new data are needed.
Define the gap precisely. A gap such as "we don't know enough about our customers" is too vague to guide research. A specified gap, in Merton's sense, might be "we do not know why customers aged under 30 cancel their subscriptions within three months". Precision makes it possible to design a study that answers the question.
Link gaps to decisions. Ask which decisions depend on each gap and how the decision would change if the answer were known. Gaps that would not change any decision are rarely worth closing.
Consider who is missing. Many gaps concern groups that are underrepresented in existing data. Filling them may require special sampling or methods.
Check for false knowledge. Assumptions that "everyone knows" can be tested cheaply. A small survey that confirms or overturns a widely held belief can be among the most valuable research an organisation commissions.
Record what remains unknown. Every study leaves some questions unanswered. Recording them, with notes on why they matter, creates an agenda for future research and prevents the same gaps from being rediscovered repeatedly.
Conclusion
Knowledge gaps come in different forms: recognised questions not yet answered, questions no one has thought to ask and beliefs that are wrong but widely held. Systematic reviews, evidence maps, the challenging of assumptions, attention to weak signals and the involvement of those affected all help to find them. Because research effort is often wasted on questions that matter little or that have already been answered, prioritising gaps is as important as finding them. Explicit criteria, value-of-information analysis and structured judgement help direct effort where it can change decisions and improve outcomes. For research organisations and their clients, the disciplined identification of what is unknown is the first step towards knowledge worth having.
References
Alvesson, M., & Sandberg, J. (2011). Generating research questions through problematization. Academy of Management Review, 36(2), 247–271. https://doi.org/10.5465/amr.2009.0188
Ansoff, H. I. (1975). Managing strategic surprise by response to weak signals. California Management Review, 18(2), 21–33. https://doi.org/10.2307/41164635
Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., Gülmezoglu, A. M., Howells, D. W., Ioannidis, J. P. A., & Oliver, S. (2014). How to increase value and reduce waste when research priorities are set. The Lancet, 383(9912), 156–165. https://doi.org/10.1016/S0140-6736(13)62229-1
Chalmers, I., & Glasziou, P. (2009). Avoidable waste in the production and reporting of research evidence. The Lancet, 374(9683), 86–89. https://doi.org/10.1016/S0140-6736(09)60329-9
Claxton, K. (1999). The irrelevance of inference: A decision-making approach to the stochastic evaluation of health care technologies. Journal of Health Economics, 18(3), 341–364. https://doi.org/10.1016/S0167-6296(98)00039-3
Claxton, K. P., & Sculpher, M. J. (2006). Using value of information analysis to prioritise health research: Some lessons from recent UK experience. PharmacoEconomics, 24(11), 1055–1068. https://doi.org/10.2165/00019053-200624110-00003
Cowan, K., & Oliver, S. (2021). The James Lind Alliance guidebook (Version 10). James Lind Alliance.
Day, G. S., & Schoemaker, P. J. H. (2006). Peripheral vision: Detecting the weak signals that will make or break your company. Harvard Business School Press.
Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins, J. P. T., Mavergames, C., & Gruen, R. L. (2014). Living systematic reviews: An emerging opportunity to narrow the evidence-practice gap. PLoS Medicine, 11(2), Article e1001603. https://doi.org/10.1371/journal.pmed.1001603
Firestein, S. (2012). Ignorance: How it drives science. Oxford University Press.
Gross, M. (2010). Ignorance and surprise: Science, society, and ecological design. MIT Press.
Higgins, J. P. T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (Eds.). (2019). Cochrane handbook for systematic reviews of interventions (2nd ed.). Wiley. https://doi.org/10.1002/9781119536604
Hubbard, D. W. (2014). How to measure anything: Finding the value of "intangibles" in business (3rd ed.). Wiley.
Huber, G. P. (1991). Organizational learning: The contributing processes and the literatures. Organization Science, 2(1), 88–115. https://doi.org/10.1287/orsc.2.1.88
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), Article e124. https://doi.org/10.1371/journal.pmed.0020124
Kahneman, D. (2011). Thinking, fast and slow. Farrar, Straus and Giroux.
Keisler, J. M., Collier, Z. A., Chu, E., Sinatra, N., & Linkov, I. (2014). Value of information analysis: The state of application. Environment Systems and Decisions, 34(1), 3–23. https://doi.org/10.1007/s10669-013-9439-4
Klein, G. (2007). Performing a project premortem. Harvard Business Review, 85(9), 18–19.
Merton, R. K. (1987). Three fragments from a sociologist's notebooks: Establishing the phenomenon, specified ignorance, and strategic research materials. Annual Review of Sociology, 13, 1–28. https://doi.org/10.1146/annurev.so.13.080187.000245
Miake-Lye, I. M., Hempel, S., Shanman, R., & Shekelle, P. G. (2016). What is an evidence map? A systematic review of published evidence maps and their definitions, methods, and products. Systematic Reviews, 5, Article 28. https://doi.org/10.1186/s13643-016-0204-x
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716
Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Blackwell. https://doi.org/10.1002/9780470754887
Proctor, R. N., & Schiebinger, L. (Eds.). (2008). Agnotology: The making and unmaking of ignorance. Stanford University Press.
Robinson, K. A., Saldanha, I. J., & McKoy, N. A. (2011). Development of a framework to identify research gaps from systematic reviews. Journal of Clinical Epidemiology, 64(12), 1325–1330. https://doi.org/10.1016/j.jclinepi.2011.06.009
Sandberg, J., & Alvesson, M. (2011). Ways of constructing research questions: Gap-spotting or problematization? Organization, 18(1), 23–44. https://doi.org/10.1177/1350508410372151
Smithson, M. (1989). Ignorance and uncertainty: Emerging paradigms. Springer. https://doi.org/10.1007/978-1-4612-3628-3
Snilstveit, B., Vojtkova, M., Bhavsar, A., Stevenson, J., & Gaarder, M. (2016). Evidence & gap maps: A tool for promoting evidence informed policy and strategic research agendas. Journal of Clinical Epidemiology, 79, 120–129. https://doi.org/10.1016/j.jclinepi.2016.05.015
Tetlock, P. E., & Gardner, D. (2015). Superforecasting: The art and science of prediction. Crown.
Viergever, R. F., Olifson, S., Ghaffar, A., & Terry, R. F. (2010). A checklist for health research priority setting: Nine common themes of good practice. Health Research Policy and Systems, 8, Article 36. https://doi.org/10.1186/1478-4505-8-36