Skip to content

Intelligence · Organisational intelligence

Feedback Mechanisms from Consumers to Improve Intelligence Knowledge Relevance

Intelligence is only useful if it meets the needs of the people who use it, yet intelligence teams often receive little feedback. This article explains why feedback from intelligence consumers matters, why it is hard to obtain, what can be learned from research on customer feedback, and which practical mechanisms help keep intelligence relevant.

ModQu research article · 2,027 words · 21 references (APA 7th edition)

Download PDF

In intelligence work, the people who receive and use intelligence products are often called consumers, or customers. They may be ministers and military commanders, or, in a business, executives, product managers and sales leaders. Intelligence exists to serve them. Yet one of the most persistent complaints in the intelligence literature is that analysts rarely learn whether their work was read, understood or useful. Without that knowledge, intelligence teams drift: they keep producing what they have always produced, answer questions nobody is asking and miss the ones that matter.

This article explains why feedback from consumers matters, why it is so often missing, what intelligence teams can learn from research on customer feedback in other fields, and which practical mechanisms help keep intelligence knowledge relevant.

Why feedback matters

Sherman Kent, writing at the start of the modern American intelligence system, argued that intelligence must be close enough to policy to know what is needed, but far enough to remain objective (Kent, 1949). Kent worried that without guidance from users, intelligence would become irrelevant; without independence, it would become biased. Feedback is the mechanism that lets intelligence stay close to user needs without abandoning its independence.

The textbook intelligence cycle includes a feedback step: consumers react to products, and their reactions shape new requirements. Lowenthal (2022) observes that in practice this step is the weakest part of the cycle. Consumers are busy, rarely say what they found useful, and often do not know what they need until they see it. Hulnick (2006) goes further, arguing that the cycle misrepresents how intelligence really works, since requirements are often vague and analysts frequently decide for themselves what to produce. Contributors to Phythian (2013) make similar points: the cycle is a useful teaching device but a poor description of practice, and its feedback loop is often missing altogether.

The consequences are serious. Davis (2006), drawing on long experience of relations between analysts and policymakers, notes that analysts who do not understand what users need tend to write for each other, producing products that are technically sound but poorly aimed. Treverton (2003) argues that in an age of abundant information, intelligence must compete for the attention of its consumers; products that do not meet their needs will simply be ignored in favour of other sources.

Why feedback is hard to obtain

Several features of intelligence work make feedback especially scarce.

Consumers are busy. Senior decision-makers have little time and many demands on it. Giving feedback on a report is low on their list.

Silence is ambiguous. When no response comes, the analyst cannot tell whether the product was read and found useful, read and found useless, or never read at all.

Outcomes are delayed and uncertain. The value of an assessment may only become clear months later, if at all, and many other factors affect the decision it informed.

Power is unequal. Analysts may hesitate to ask senior consumers for criticism, and consumers may not want to admit that they did not read a product.

Independence creates distance. Intelligence organisations deliberately keep some distance from policy to protect objectivity. Betts (2007) describes this tension as a permanent feature of intelligence: too close, and analysis is politicised; too far, and it is irrelevant. Distance can easily become an excuse for not asking.

What customer feedback research teaches

Intelligence is a service, and research on how service organisations collect and use customer feedback offers useful lessons.

Exit and voice. Hirschman (1970) distinguished two responses to declining quality. Dissatisfied customers can exit, by switching to another supplier, or use voice, by complaining in the hope of improvement. Exit gives the organisation little information; voice tells it what is wrong. Intelligence consumers who are dissatisfied usually exit quietly: they stop reading, rely on their own staff or turn to news and consultants. An intelligence team that makes voice easy and welcome learns far more than one that waits for consumers to drift away.

Expectations shape satisfaction. Oliver (1980) showed that satisfaction depends on how performance compares with expectations. A consumer who expected a short answer and received a long report may be dissatisfied, even if the report is excellent. Teams should therefore agree expectations about format, length, timing and confidence at the start of a task, and check whether they were met.

Dimensions of quality. Parasuraman et al. (1988) identified dimensions along which customers judge service quality, including reliability, responsiveness, assurance and empathy. These map well onto intelligence: consumers value products that are accurate and dependable, delivered when needed, written with evident expertise and aimed at their particular concerns.

Measuring satisfaction. National satisfaction indices, such as the one described by Fornell et al. (1996), show that satisfaction can be measured systematically and linked to loyalty. Reichheld (2003) popularised a single-question measure, the Net Promoter Score, asking how likely customers are to recommend a company. Keiningham et al. (2007), however, found that this single measure predicted growth no better than other satisfaction measures. The lesson for intelligence teams is that a single score is a crude guide; it is better to ask several specific questions and to listen to the reasons behind the answers.

Complaints are opportunities. Tax et al. (1998) found that how an organisation handles a complaint strongly affects whether the customer remains loyal, and that fair procedures and genuine responses matter as much as the outcome. Smith et al. (1999) showed that customers judge recovery from a service failure on whether the response matches the seriousness of the problem. When an intelligence product misses the mark, a prompt and honest response can strengthen the relationship.

Voluntary feedback is biased. Hu et al. (2009) showed that online product reviews follow a J-shaped distribution: people with very positive or very negative experiences are more likely to write reviews than those with moderate views. Anderson (1998) found that both very satisfied and very dissatisfied customers talk more about their experiences than those in the middle. Feedback that arrives unprompted is therefore unrepresentative. Intelligence teams that rely only on occasional praise or complaints will get a distorted picture.

Informal conversation is informative. Godes and Mayzlin (2004) showed that online conversations between customers can be used to measure word of mouth. Inside organisations, informal comments about intelligence products, in meetings or corridors, can be a valuable signal if someone listens and records them.

Five-step feedback loop: agree expectations, deliver product, gather feedback, analyse patterns and adjust requirements, with the cycle repeating.
Figure 1. A feedback loop for intelligence products. Feedback is gathered deliberately, not left to chance, and used to change what the team produces. Based on Lowenthal (2022), Oliver (1980) and Hirschman (1970); diagram by ModQu.

Practical feedback mechanisms

Intelligence teams can use several mechanisms, ideally in combination.

Agree the task. Before starting significant work, meet the consumer to agree the question, the decision it supports, the deadline, the format and the level of detail. Recording these terms gives a standard against which to judge the product later.

Short feedback requests. Attach a brief request to products: two or three questions, such as whether the product was useful, whether it arrived in time and what was missing. Keep it short enough to complete in a minute.

Follow-up conversations. For important products, ask for a brief conversation a few days after delivery. Kerbel and Olcott (2010) argue that the most valuable analysis emerges from continuing dialogue with consumers, in which analysts and users make sense of a situation together, rather than from finished products delivered from a distance. Conversation also reveals needs that consumers would not think to write down.

Periodic reviews. Every six or twelve months, interview key consumers about the team's overall performance: which products they use, which they ignore, what has changed in their priorities and what they would like more or less of.

Usage data. Where products are delivered electronically, track which are opened, how long readers spend on them and which are forwarded. Usage data does not show whether a product helped, but it does show what is read.

Embedded analysts. Placing an analyst within a consumer's team, at least part time, gives continuing informal feedback and understanding of how decisions are actually made.

Outcome tracking. Record key judgements and compare them with what happened. This is feedback from reality rather than from consumers, and it complements what consumers say. Marrin (2012) argues that intelligence analysis should build an evidence base about which practices work; outcome tracking is one way to do so.

Questions worth asking consumers

The questions asked shape the answers received. Useful questions are specific and tied to decisions:

  1. Which decision did this product help with, if any?
  2. What did you do differently, or more confidently, because of it?
  3. What was missing that you needed?
  4. Was it the right length, and did it arrive in time?
  5. What question should we be working on next?

Questions about decisions and actions are harder to answer politely than questions about satisfaction, which makes the answers more informative. A consumer who cannot name any decision the product helped with is telling the team something important, even if they rate the product highly.

Using feedback well

Collecting feedback is the easier part. Using it well requires some discipline.

First, look for patterns rather than reacting to every comment. One consumer's complaint about length may reflect personal preference; five consumers' complaints suggest a real problem.

Second, distinguish feedback about form from feedback about substance. Consumers may dislike a judgement because it is unwelcome, not because it is wrong. Fingar (2011) stresses that intelligence must sometimes deliver unwelcome messages; feedback should improve relevance and clarity, not push analysts towards telling consumers what they want to hear.

Third, close the loop. Tell consumers what changed as a result of their feedback. People who see that their comments led to improvements are more willing to give feedback again.

Fourth, share feedback within the team. Analysts improve fastest when they see how their products were received, both praise and criticism. A short monthly discussion of what consumers said, and what the team will change in response, turns individual comments into shared learning and makes feedback part of the team's normal routine rather than an occasional exercise.

Common mistakes

A business illustration

A software company's market intelligence team produces a monthly competitor report of around thirty pages. It assumes the report is valued because no one has complained. A new team leader introduces a three-question feedback request and interviews eight senior consumers. The results are uncomfortable: most read only the summary, two have stopped reading altogether and rely on their sales teams instead, and all want faster warning of competitors' pricing changes.

The team changes its approach. It replaces the monthly report with a two-page summary and a short alert service for significant competitor moves. It agrees with each product director the three or four questions that matter most to them, and an analyst joins the product planning meeting each month. After six months, a repeat survey shows that consumers find the products more useful, and two directors who had stopped reading have returned. The team now reviews its feedback each quarter and reports to consumers what it has changed.

Conclusion

Feedback from consumers is the mechanism that keeps intelligence relevant. Without it, intelligence teams work in the dark, producing products that may be accurate but are poorly aimed. Feedback is scarce in intelligence work because consumers are busy, silence is ambiguous and distance from users is built into the profession. Research on customer feedback shows that dissatisfied users tend to leave quietly, that satisfaction depends on expectations and that unprompted feedback is unrepresentative. By agreeing expectations, asking short questions, holding regular conversations, tracking outcomes and showing consumers what changed, intelligence teams can keep their knowledge aimed at the decisions that matter.

References

Anderson, E. W. (1998). Customer satisfaction and word of mouth. Journal of Service Research, 1(1), 5–17. https://doi.org/10.1177/109467059800100102

Betts, R. K. (2007). Enemies of intelligence: Knowledge and power in American national security. Columbia University Press.

Davis, J. (2006). Intelligence analysts and policymakers: Benefits and dangers of tensions in the relationship. Intelligence and National Security, 21(6), 999–1021. https://doi.org/10.1080/02684520601046325

Fingar, T. (2011). Reducing uncertainty: Intelligence analysis and national security. Stanford University Press.

Fornell, C., Johnson, M. D., Anderson, E. W., Cha, J., & Bryant, B. E. (1996). The American Customer Satisfaction Index: Nature, purpose, and findings. Journal of Marketing, 60(4), 7–18. https://doi.org/10.1177/002224299606000403

Godes, D., & Mayzlin, D. (2004). Using online conversations to study word-of-mouth communication. Marketing Science, 23(4), 545–560. https://doi.org/10.1287/mksc.1040.0071

Hirschman, A. O. (1970). Exit, voice, and loyalty: Responses to decline in firms, organizations, and states. Harvard University Press.

Hu, N., Zhang, J., & Pavlou, P. A. (2009). Overcoming the J-shaped distribution of product reviews. Communications of the ACM, 52(10), 144–147. https://doi.org/10.1145/1562764.1562800

Hulnick, A. S. (2006). What's wrong with the intelligence cycle. Intelligence and National Security, 21(6), 959–979. https://doi.org/10.1080/02684520601046291

Keiningham, T. L., Cooil, B., Andreassen, T. W., & Aksoy, L. (2007). A longitudinal examination of net promoter and firm revenue growth. Journal of Marketing, 71(3), 39–51. https://doi.org/10.1509/jmkg.71.3.039

Kent, S. (1949). Strategic intelligence for American world policy. Princeton University Press.

Kerbel, J., & Olcott, A. (2010). Synthesizing with clients, not analyzing for customers. Studies in Intelligence, 54(4), 11–27.

Lowenthal, M. M. (2022). Intelligence: From secrets to policy (9th ed.). CQ Press.

Marrin, S. (2012). Improving intelligence analysis: Bridging the gap between scholarship and practice. Routledge.

Oliver, R. L. (1980). A cognitive model of the antecedents and consequences of satisfaction decisions. Journal of Marketing Research, 17(4), 460–469. https://doi.org/10.1177/002224378001700405

Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1988). SERVQUAL: A multiple-item scale for measuring consumer perceptions of service quality. Journal of Retailing, 64(1), 12–40.

Phythian, M. (Ed.). (2013). Understanding the intelligence cycle. Routledge. https://doi.org/10.4324/9780203558478

Reichheld, F. F. (2003). The one number you need to grow. Harvard Business Review, 81(12), 46–54.

Smith, A. K., Bolton, R. N., & Wagner, J. (1999). A model of customer satisfaction with service encounters involving failure and recovery. Journal of Marketing Research, 36(3), 356–372. https://doi.org/10.1177/002224379903600305

Tax, S. S., Brown, S. W., & Chandrashekaran, M. (1998). Customer evaluations of service complaint experiences: Implications for relationship marketing. Journal of Marketing, 62(2), 60–76. https://doi.org/10.1177/002224299806200205

Treverton, G. F. (2003). Reshaping national intelligence for an age of information. Cambridge University Press.