Insights · Tools and the future
The Economics of Insight: Who Pays for Knowledge and Why
The economics of producing and buying insight, including the value of information for decisions, why knowledge is costly to produce but cheap to copy, the different ways insight is funded and sold, how organisations decide how much to spend, and why much research is never used.
Insight costs money. Surveys, panels, interviews, data platforms, analysts and consultants are significant expenses, and organisations must decide how much to spend, on what and with whom. Yet insight is an unusual kind of good. Its value depends on decisions not yet made, it is expensive to produce but cheap to copy, its quality is hard to judge before it is used, and much of it is never used at all. Economics offers useful tools for thinking about these features. This article explains the economics of insight: what it is worth, who pays for it and why, and how organisations can spend on it more wisely.
Information as an economic good
Economists have long recognised that information behaves differently from ordinary goods. Stigler (1961) analysed the costs and benefits of searching for information, such as the best price in a market, and showed that people rationally stop searching when the expected gain from further search falls below its cost. Arrow (1962) identified a fundamental difficulty in selling information: a buyer cannot judge its value without knowing it, but once the seller has revealed it, the buyer has it without paying. Arrow also noted that once information exists, sharing it costs little, so its production may be undersupplied by markets.
Shapiro and Varian (1998) described information goods as having high fixed costs of production and very low costs of reproduction, which shapes how they are priced and sold. A survey of 5,000 consumers costs a great deal to run, but its results can be copied and distributed to many clients at almost no extra cost. This cost structure explains why research suppliers sell the same studies to many clients, why they often price by the value to the buyer rather than the cost of supply, and why buyers who commission exclusive research pay far more than those who share it. Romer (1990) emphasised a related property: knowledge is "non-rival", meaning one person's use does not reduce its availability to others, which makes it a powerful driver of growth but difficult to own.
Insight is also an "experience good" whose quality is hard to judge in advance. Akerlof (1970) showed how markets can deteriorate when buyers cannot distinguish high-quality from low-quality goods, because sellers of poor goods can pass them off as good, and buyers respond by paying only average prices, which drives good sellers out. Research buyers face this problem: a cheap survey and a well-designed one may look similar on a proposal, and the difference may become apparent only when decisions based on the results go wrong.
The value of information for decisions
The most useful economic idea for insight work is the value of information. Raiffa and Schlaifer (1961) developed the statistical decision theory behind it, and Howard (1966) described "information value theory" as a way of calculating how much a decision maker should be willing to pay for information before making a decision. The key point is that information has value only if it might change a decision. If the organisation would take the same action whatever the research showed, the research is worth nothing for that decision, however interesting its findings.
A simple example shows the calculation. Suppose a company is deciding whether to launch a product. It estimates a 50% chance of success, which would bring a profit of 10 million, and a 50% chance of failure, which would bring a loss of 6 million. If it launches without further information, the expected value is 0.5 × 10 − 0.5 × 6 = 2 million. If it could know in advance whether the product would succeed, it would launch only when it would succeed, and the expected value would be 0.5 × 10 = 5 million. The difference, 3 million, is the expected value of perfect information: the most it would be worth paying for research that could predict success with certainty.
Real research is never perfect, so its value is lower, and depends on how accurately it predicts the outcome. But the logic is general. Research is most valuable when:
- The decision is uncertain. If the outcome is already almost certain, research adds little.
- The stakes are high. The bigger the gains and losses, the more it is worth reducing uncertainty.
- The research could change the decision. If the organisation is committed to a course of action regardless, research has little decision value.
- The research is accurate enough to distinguish the outcomes that matter.
A useful exercise before commissioning research is to ask the decision maker what result would lead them to choose differently. If no plausible result would change the decision, the research may still have value for learning, but it should not be justified as decision support, and its budget should reflect that.
Hubbard (2014) argued that organisations often spend most on measuring things that matter least, and little on the most uncertain and consequential variables, and proposed that the value of information be calculated to guide what to measure. Hubbard also noted that the first small amount of information often reduces uncertainty the most, so modest, quick research can be worth much more than its cost.
Who pays for insight
Because insight is costly to produce and cheap to copy, it is funded in several different ways.
Commissioned research. An organisation pays for research that it alone owns. This makes sense when the insight concerns its own customers or products, or when exclusive knowledge gives it an advantage. Barney (1991) argued that resources yield lasting competitive advantage when they are valuable, rare and hard to imitate, and proprietary insight can meet these conditions.
Syndicated research. Many organisations pay for the same study, such as a market tracking report or retail audit, sharing its fixed costs. This exploits the low cost of reproduction: each buyer pays less than the cost of commissioning alone, but none gains exclusive knowledge. Bakos and Brynjolfsson (1999) showed that bundling many information goods together, as syndicated services often do, can be profitable for sellers and efficient for buyers.
Public research. Governments, universities and foundations fund research whose results are freely available, such as official statistics and academic studies. Samuelson (1954) described public goods, which are non-rival and non-excludable, and showed that markets tend to undersupply them. Official statistics on population, prices and spending are a foundation on which much commercial insight builds. They are often underused by businesses, which pay for commercial studies that duplicate information already published at no charge.
Data exchange. Many digital services are free to users, who in effect pay with their data and attention. Varian (2010) described how computer-mediated transactions allow companies to observe and analyse behaviour at low cost, generating insight as a by-product of providing a service. Bergemann and Bonatti (2019) reviewed how markets for information and data operate, including the ways in which data collected from users is sold or used to target them.
Sponsored research. Some research and reports are funded by sponsors and offered free. This lowers the cost to readers but may compromise independence, since sponsors have an interest in the conclusions.
Why insight is often wasted
Organisations spend heavily on research that is never used. Menon and Varadarajan (1992) distinguished between using research to take action, to inform general understanding, and to justify decisions already made. Deshpandé and Zaltman (1982) studied the use of market research by managers and found that research was more likely to be used when it was technically sound, when its findings were not surprising enough to be dismissed, and when researchers and managers had built trust. Moorman (1995) found that organisational culture shaped whether market information was used.
Cohen and Levinthal (1990) introduced the concept of "absorptive capacity", an organisation's ability to recognise the value of new external knowledge, absorb it and apply it, and argued that this capacity depends on the organisation's existing knowledge. An organisation that buys research without the expertise to interpret it gains little. Spending on research without spending on the people who use it is therefore a common source of waste. Grossman and Stiglitz (1980) showed that if information were perfectly reflected in market prices, no one would have an incentive to pay for it, which helps explain why insight has value only when it is not already widely known, and why widely available reports offer less advantage than proprietary analysis.
Buying research well
Because research quality is hard to judge in advance, buyers need ways to reduce the risk of paying for poor work. Several practices help.
Specify the decision, not only the method. A brief that explains the decision the research will inform, and what would change it, allows suppliers to propose designs that fit, and allows the buyer to judge whether a proposal addresses the question.
Ask how quality will be checked. Good suppliers can explain how they will recruit respondents, check data quality, test questions and handle uncertainty. Vague answers are a warning sign.
Pay for what matters. Sample size, fieldwork quality and analytical expertise usually matter more than presentation. A polished report built on a poorly designed survey is worth less than a plain one built on a sound design.
Build relationships. Deshpandé and Zaltman (1982) found that trust between researchers and managers increased the use of research. Long-term relationships with a small number of suppliers allow them to learn the business and allow buyers to judge their quality from experience, reducing the problem of judging an experience good in advance.
Reuse and combine. Much insight can be gained by analysing existing data, combining previous studies or adding a few questions to an existing survey, at a fraction of the cost of a new study.
Measuring the return on insight
Organisations often struggle to show the return on their insight spending, because the benefit lies in decisions that were improved or mistakes that were avoided, which are hard to observe. Some evidence is available at the level of firms. Brynjolfsson et al. (2011) found that firms that reported making decisions based on data had higher output and productivity than would be expected from their other investments. Within an organisation, a practical approach is to record, for major research projects, the decision it informed, the option chosen and whether the research changed it, and to review these records periodically. Over time, this shows which kinds of research change decisions and which do not, and allows spending to shift towards the former.
The economics of experiments
Online experimentation has changed the economics of some insight. Kohavi et al. (2020) described how large online businesses run thousands of experiments at very low marginal cost, testing changes on a share of users. When experiments are cheap, it often makes sense to test rather than research in advance. But the value of experimentation also depends on the distribution of possible effects. Azevedo et al. (2020) analysed a large set of experiments at a technology company and found that the distribution of effects was "fat-tailed": most changes had small effects, but a few had very large ones. In such conditions, they showed, it can pay to run many small experiments in search of the rare big winners rather than a few large ones.
Spending wisely
Several principles follow for organisations deciding how much to spend on insight:
- Start from the decision. Fund research in proportion to the stakes and uncertainty of the decisions it informs, and ask whether it could change them.
- Buy shared knowledge cheaply, invest in proprietary knowledge. Use syndicated and public sources for general market understanding; reserve commissioned research for questions where exclusive insight creates advantage.
- Value speed and modest research. Quick, inexpensive research often captures much of the available value.
- Build absorptive capacity. Invest in people who can interpret and apply research, not only in research itself.
- Judge quality, not price. Ask how research will be designed and checked, since quality is hard to see but determines value.
- Track use. Record which research informed which decisions, and review whether it was worth its cost.
Key takeaways
Insight is an unusual economic good: costly to produce, cheap to copy, hard to judge before use and valuable only if it might change a decision. The value of information depends on uncertainty, stakes and the chance that research will change a decision, and can be estimated before research is commissioned. Insight is funded in several ways, from commissioned and syndicated research to public statistics, data exchange and sponsorship, each with trade-offs. Much research is wasted because organisations lack the capacity or culture to use it. Spending wisely means starting from the decision, distinguishing shared from proprietary knowledge, valuing quick research, building the capacity to use insight and tracking whether it was used.
References
Akerlof, G. A. (1970). The market for "lemons": Quality uncertainty and the market mechanism. The Quarterly Journal of Economics, 84(3), 488–500. https://doi.org/10.2307/1879431
Arrow, K. J. (1962). Economic welfare and the allocation of resources for invention. In National Bureau of Economic Research (Ed.), The rate and direction of inventive activity: Economic and social factors (pp. 609–626). Princeton University Press.
Azevedo, E. M., Deng, A., Montiel Olea, J. L., Rao, J., & Weyl, E. G. (2020). A/B testing with fat tails. Journal of Political Economy, 128(12), 4614–4672. https://doi.org/10.1086/710607
Bakos, Y., & Brynjolfsson, E. (1999). Bundling information goods: Pricing, profits, and efficiency. Management Science, 45(12), 1613–1630. https://doi.org/10.1287/mnsc.45.12.1613
Barney, J. (1991). Firm resources and sustained competitive advantage. Journal of Management, 17(1), 99–120. https://doi.org/10.1177/014920639101700108
Bergemann, D., & Bonatti, A. (2019). Markets for information: An introduction. Annual Review of Economics, 11, 85–107. https://doi.org/10.1146/annurev-economics-080315-015439
Brynjolfsson, E., Hitt, L. M., & Kim, H. H. (2011). Strength in numbers: How does data-driven decisionmaking affect firm performance? (Working paper). SSRN. https://doi.org/10.2139/ssrn.1819486
Cohen, W. M., & Levinthal, D. A. (1990). Absorptive capacity: A new perspective on learning and innovation. Administrative Science Quarterly, 35(1), 128–152. https://doi.org/10.2307/2393553
Deshpandé, R., & Zaltman, G. (1982). Factors affecting the use of market research information: A path analysis. Journal of Marketing Research, 19(1), 14–31. https://doi.org/10.1177/002224378201900102
Grossman, S. J., & Stiglitz, J. E. (1980). On the impossibility of informationally efficient markets. The American Economic Review, 70(3), 393–408.
Howard, R. A. (1966). Information value theory. IEEE Transactions on Systems Science and Cybernetics, 2(1), 22–26. https://doi.org/10.1109/TSSC.1966.300074
Hubbard, D. W. (2014). How to measure anything: Finding the value of "intangibles" in business (3rd ed.). Wiley.
Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. https://doi.org/10.1017/9781108653985
Menon, A., & Varadarajan, P. R. (1992). A model of marketing knowledge use within firms. Journal of Marketing, 56(4), 53–71. https://doi.org/10.1177/002224299205600405
Moorman, C. (1995). Organizational market information processes: Cultural antecedents and new product outcomes. Journal of Marketing Research, 32(3), 318–335. https://doi.org/10.1177/002224379503200307
Raiffa, H., & Schlaifer, R. (1961). Applied statistical decision theory. Division of Research, Graduate School of Business Administration, Harvard University.
Romer, P. M. (1990). Endogenous technological change. Journal of Political Economy, 98(5, Part 2), S71–S102. https://doi.org/10.1086/261725
Samuelson, P. A. (1954). The pure theory of public expenditure. The Review of Economics and Statistics, 36(4), 387–389. https://doi.org/10.2307/1925895
Shapiro, C., & Varian, H. R. (1998). Information rules: A strategic guide to the network economy. Harvard Business School Press.
Stigler, G. J. (1961). The economics of information. Journal of Political Economy, 69(3), 213–225. https://doi.org/10.1086/258464
Varian, H. R. (2010). Computer mediated transactions. American Economic Review, 100(2), 1–10. https://doi.org/10.1257/aer.100.2.1