Intelligence · Organisational intelligence
Measuring the Value of Intelligence
Intelligence functions are often asked to prove their worth, and often struggle to do so. This article explains why the value of intelligence is hard to measure, the main approaches from simple activity counts to formal value-of-information analysis, and how to build a practical measurement system.
Sooner or later, every intelligence function is asked what it is worth. Budgets are limited, and managers want to know whether money spent on research, analysis and monitoring pays for itself. The question is fair but hard to answer. Intelligence rarely produces revenue directly. Its value lies in better decisions, avoided mistakes and earlier warning, all of which are difficult to observe and harder to attribute. Yet functions that cannot show their value tend to lose resources, so measurement matters.
Why value is hard to measure
Several features of intelligence make measurement difficult:
- Value is indirect. Intelligence informs decisions, and decisions produce results. Many other factors affect those results.
- Success is often invisible. A warning that leads to a problem being avoided leaves nothing to see. The crisis that did not happen cannot be counted.
- Timing is long. The value of a strategic assessment may become clear only years later.
- Counterfactuals are unknown. It is rarely possible to know what would have happened without the intelligence.
Hubbard (2014) argues that such difficulties are often overstated. Anything that matters to decisions has observable consequences, and those consequences can be measured, at least approximately. The aim is not perfect precision but reducing uncertainty enough to make better decisions about the intelligence function itself.
Four levels of measurement
Measures of intelligence value range from easy but shallow to hard but meaningful. Figure 1 sets out four levels.
Activity measures count what the function produces: reports issued, requests answered, sources monitored. They are easy to collect but say nothing about usefulness. They also carry a risk. Kerr (1975) described the folly of rewarding one behaviour while hoping for another; a function rewarded for the number of reports will produce more reports, not better ones. Campbell (1979) warned that the more a quantitative indicator is used for decisions, the more it will be subject to corruption and distortion. Muller (2018) documents how excessive reliance on such metrics damages performance in many fields.
User satisfaction measures ask decision-makers whether intelligence was useful, through ratings, feedback or the rate of repeat requests. They are closer to value but can be misleading: users may rate highly the products that confirm their views.
Decision influence measures record which decisions intelligence informed and how. A simple log, noting each significant decision, the intelligence provided and whether it changed the outcome, can be built up over time. Herring (1999) recommended linking every intelligence product to a key intelligence topic and, through it, to a decision, which makes such tracking possible.
Outcome value measures estimate the money saved or earned, risks avoided or forecasts improved. They are the most meaningful and the hardest to obtain. They usually require an estimate of what would have happened without the intelligence, which should be made cautiously and agreed with the decision-maker involved, so that the figure is credible to others in the organisation.
Value of information
Decision analysis offers a formal way to put a value on intelligence. Howard (1966) developed information value theory, and Raiffa (1968) set out the methods of decision analysis on which it rests. The idea is that information has value only if it might change a decision. Its value equals the improvement in the expected result of the decision when the information is available, compared with when it is not.
Figure 2 works through a simple example.
The example shows three useful points. First, the value of perfect information sets an upper limit on what any intelligence could be worth for that decision. Second, imperfect intelligence is worth less, in proportion to its accuracy. Third, if the decision would be the same whatever the intelligence said, the intelligence has no decision value at all, however interesting it is. Hubbard (2014) argues that such calculations, even with rough estimates, often show that organisations measure the wrong things and neglect the uncertainties that matter most.
Measuring accuracy
For forecasts and estimates, accuracy can be measured directly by recording predictions and comparing them with outcomes. Brier (1950) devised a score for probability forecasts that rewards both accuracy and appropriate confidence. Mellers et al. (2014) and Tetlock and Gardner (2015) showed in large forecasting tournaments that scoring forecasts in this way allows organisations to identify skilled forecasters, test methods and improve over time. An intelligence function that records its probability judgements and scores them builds a direct measure of the quality of its analysis.
Building a measurement system
Kaplan and Norton (1996) argued that organisations should measure performance through a balanced set of measures, not a single figure. Neely, Gregory and Platts (1995) reviewed the design of performance measurement systems and emphasised that measures should follow from strategy. For an intelligence function, a balanced system might include:
- A small number of activity measures, used for management, not as targets.
- Regular user feedback, ideally including open questions about what was useful.
- A decision log, recording significant decisions supported and their outcomes.
- Forecast scoring, for any probability judgements made.
- Occasional case studies, describing in detail how intelligence affected a significant decision.
Calof and Wright (2008) note that the competitive intelligence field has long struggled to demonstrate value, and that case studies of specific decisions are often more persuasive to executives than aggregate statistics. Gilad (2004) argues that the clearest value of intelligence lies in early warning that allows a company to act before rivals.
An illustration
The intelligence team at a regional bank is asked by the finance director to justify its budget. Its usual report lists the number of briefings produced and the topics covered. This time, the team reviews the past year's decision log. It identifies three decisions where its work made a clear difference. A warning about a competitor's planned branch closures allowed the bank to recruit staff and target customers in the affected towns, bringing in new accounts worth a measurable sum. An assessment of a proposed partnership flagged regulatory risks that led the board to renegotiate terms. And a forecast of mortgage demand, scored against the outcome, proved more accurate than the previous method, allowing better staffing.
The team presents these three cases with conservative estimates of their value, alongside user feedback and its forecast scores. The total estimated value is several times the team's annual cost. The finance director, who had expected a list of activities, agrees to maintain the budget and asks for the decision log to be shared quarterly.
Common mistakes
- Reporting only activity. Counts of reports and requests invite the question "So what?"
- Claiming all the credit. Intelligence is one input among many. Conservative, honest estimates are more credible than inflated ones.
- Measuring too late. Records of decisions and forecasts must be kept as they happen; reconstructing them later is unreliable.
- Turning measures into targets. Once a number becomes a target, people work to the number rather than to the purpose.
- Ignoring failures. Cases where intelligence was wrong or ignored are as informative as successes and make the overall account more credible.
Conclusion
The value of intelligence is real but indirect, and measuring it requires effort. Activity counts are easy but misleading; satisfaction is useful but partial. Tracking decisions, scoring forecasts and, where possible, estimating the value of information give a fuller picture. A function that measures its value honestly not only secures its resources but learns where it is most useful.
References
Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3. https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2
Calof, J. L., & Wright, S. (2008). Competitive intelligence: A practitioner, academic and inter-disciplinary perspective. European Journal of Marketing, 42(7/8), 717–730. https://doi.org/10.1108/03090560810877114
Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67–90. https://doi.org/10.1016/0149-7189(79)90048-X
Gilad, B. (2004). Early warning: Using competitive intelligence to anticipate market shifts, control risk, and create powerful strategies. AMACOM.
Herring, J. P. (1999). Key intelligence topics: A process to identify and define intelligence needs. Competitive Intelligence Review, 10(2), 4–14.
Howard, R. A. (1966). Information value theory. IEEE Transactions on Systems Science and Cybernetics, 2(1), 22–26. https://doi.org/10.1109/TSSC.1966.300074
Hubbard, D. W. (2014). How to measure anything: Finding the value of "intangibles" in business (3rd ed.). Wiley.
Kaplan, R. S., & Norton, D. P. (1996). The balanced scorecard: Translating strategy into action. Harvard Business School Press.
Kerr, S. (1975). On the folly of rewarding A, while hoping for B. Academy of Management Journal, 18(4), 769–783. https://doi.org/10.5465/255378
Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D., Atanasov, P., Swift, S. A., Murray, T., Stone, E., & Tetlock, P. E. (2014). Psychological strategies for winning a geopolitical forecasting tournament. Psychological Science, 25(5), 1106–1115. https://doi.org/10.1177/0956797614524255
Muller, J. Z. (2018). The tyranny of metrics. Princeton University Press.
Neely, A., Gregory, M., & Platts, K. (1995). Performance measurement system design: A literature review and research agenda. International Journal of Operations & Production Management, 15(4), 80–116. https://doi.org/10.1108/01443579510083622
Raiffa, H. (1968). Decision analysis: Introductory lectures on choices under uncertainty. Addison-Wesley.
Tetlock, P. E., & Gardner, D. (2015). Superforecasting: The art and science of prediction. Crown.