Skip to content

Insights · Tools and the future

The Economics of Insight: Who Pays for Knowledge and Why

The economics of producing and buying insight, including the value of information for decisions, why knowledge is costly to produce but cheap to copy, the different ways insight is funded and sold, how organisations decide how much to spend, and why much research is never used.

ModQu research article · 2,152 words · 21 references (APA 7th edition)

Download PDF

Insight costs money. Surveys, panels, interviews, data platforms, analysts and consultants are significant expenses, and organisations must decide how much to spend, on what and with whom. Yet insight is an unusual kind of good. Its value depends on decisions not yet made, it is expensive to produce but cheap to copy, its quality is hard to judge before it is used, and much of it is never used at all. Economics offers useful tools for thinking about these features. This article explains the economics of insight: what it is worth, who pays for it and why, and how organisations can spend on it more wisely.

Information as an economic good

Economists have long recognised that information behaves differently from ordinary goods. Stigler (1961) analysed the costs and benefits of searching for information, such as the best price in a market, and showed that people rationally stop searching when the expected gain from further search falls below its cost. Arrow (1962) identified a fundamental difficulty in selling information: a buyer cannot judge its value without knowing it, but once the seller has revealed it, the buyer has it without paying. Arrow also noted that once information exists, sharing it costs little, so its production may be undersupplied by markets.

Shapiro and Varian (1998) described information goods as having high fixed costs of production and very low costs of reproduction, which shapes how they are priced and sold. A survey of 5,000 consumers costs a great deal to run, but its results can be copied and distributed to many clients at almost no extra cost. This cost structure explains why research suppliers sell the same studies to many clients, why they often price by the value to the buyer rather than the cost of supply, and why buyers who commission exclusive research pay far more than those who share it. Romer (1990) emphasised a related property: knowledge is "non-rival", meaning one person's use does not reduce its availability to others, which makes it a powerful driver of growth but difficult to own.

Insight is also an "experience good" whose quality is hard to judge in advance. Akerlof (1970) showed how markets can deteriorate when buyers cannot distinguish high-quality from low-quality goods, because sellers of poor goods can pass them off as good, and buyers respond by paying only average prices, which drives good sellers out. Research buyers face this problem: a cheap survey and a well-designed one may look similar on a proposal, and the difference may become apparent only when decisions based on the results go wrong.

The value of information for decisions

The most useful economic idea for insight work is the value of information. Raiffa and Schlaifer (1961) developed the statistical decision theory behind it, and Howard (1966) described "information value theory" as a way of calculating how much a decision maker should be willing to pay for information before making a decision. The key point is that information has value only if it might change a decision. If the organisation would take the same action whatever the research showed, the research is worth nothing for that decision, however interesting its findings.

A simple example shows the calculation. Suppose a company is deciding whether to launch a product. It estimates a 50% chance of success, which would bring a profit of 10 million, and a 50% chance of failure, which would bring a loss of 6 million. If it launches without further information, the expected value is 0.5 × 10 − 0.5 × 6 = 2 million. If it could know in advance whether the product would succeed, it would launch only when it would succeed, and the expected value would be 0.5 × 10 = 5 million. The difference, 3 million, is the expected value of perfect information: the most it would be worth paying for research that could predict success with certainty.

Bar chart for a hypothetical product launch decision. The expected value of deciding now, without further information, is 2 million. The expected value with perfect information is 5 million. The difference, the value of perfect information, is 3 million, which is the maximum worth paying for research. Real research is imperfect and worth less.
Figure 1. The value of information for a hypothetical launch decision. Research can be worth at most the gain from deciding better. Calculated values, chart by ModQu, based on Howard (1966).

Real research is never perfect, so its value is lower, and depends on how accurately it predicts the outcome. But the logic is general. Research is most valuable when:

A useful exercise before commissioning research is to ask the decision maker what result would lead them to choose differently. If no plausible result would change the decision, the research may still have value for learning, but it should not be justified as decision support, and its budget should reflect that.

Hubbard (2014) argued that organisations often spend most on measuring things that matter least, and little on the most uncertain and consequential variables, and proposed that the value of information be calculated to guide what to measure. Hubbard also noted that the first small amount of information often reduces uncertainty the most, so modest, quick research can be worth much more than its cost.

Who pays for insight

Because insight is costly to produce and cheap to copy, it is funded in several different ways.

Table of five ways insight is funded: commissioned (one organisation pays for proprietary research; it owns the results and gains an advantage), syndicated (many buyers share the cost of the same study; cheaper but not exclusive), public (governments and universities fund research; results are free to all), data exchange (customers provide data in return for free services; the organisation uses it for insight), and advertising-supported (free research or reports funded by sponsors; independence may be at risk).
Figure 2. Five ways insight is funded, each with its own trade-offs. Diagram by ModQu, based on Shapiro and Varian (1998).

Commissioned research. An organisation pays for research that it alone owns. This makes sense when the insight concerns its own customers or products, or when exclusive knowledge gives it an advantage. Barney (1991) argued that resources yield lasting competitive advantage when they are valuable, rare and hard to imitate, and proprietary insight can meet these conditions.

Syndicated research. Many organisations pay for the same study, such as a market tracking report or retail audit, sharing its fixed costs. This exploits the low cost of reproduction: each buyer pays less than the cost of commissioning alone, but none gains exclusive knowledge. Bakos and Brynjolfsson (1999) showed that bundling many information goods together, as syndicated services often do, can be profitable for sellers and efficient for buyers.

Public research. Governments, universities and foundations fund research whose results are freely available, such as official statistics and academic studies. Samuelson (1954) described public goods, which are non-rival and non-excludable, and showed that markets tend to undersupply them. Official statistics on population, prices and spending are a foundation on which much commercial insight builds. They are often underused by businesses, which pay for commercial studies that duplicate information already published at no charge.

Data exchange. Many digital services are free to users, who in effect pay with their data and attention. Varian (2010) described how computer-mediated transactions allow companies to observe and analyse behaviour at low cost, generating insight as a by-product of providing a service. Bergemann and Bonatti (2019) reviewed how markets for information and data operate, including the ways in which data collected from users is sold or used to target them.

Sponsored research. Some research and reports are funded by sponsors and offered free. This lowers the cost to readers but may compromise independence, since sponsors have an interest in the conclusions.

Why insight is often wasted

Organisations spend heavily on research that is never used. Menon and Varadarajan (1992) distinguished between using research to take action, to inform general understanding, and to justify decisions already made. Deshpandé and Zaltman (1982) studied the use of market research by managers and found that research was more likely to be used when it was technically sound, when its findings were not surprising enough to be dismissed, and when researchers and managers had built trust. Moorman (1995) found that organisational culture shaped whether market information was used.

Cohen and Levinthal (1990) introduced the concept of "absorptive capacity", an organisation's ability to recognise the value of new external knowledge, absorb it and apply it, and argued that this capacity depends on the organisation's existing knowledge. An organisation that buys research without the expertise to interpret it gains little. Spending on research without spending on the people who use it is therefore a common source of waste. Grossman and Stiglitz (1980) showed that if information were perfectly reflected in market prices, no one would have an incentive to pay for it, which helps explain why insight has value only when it is not already widely known, and why widely available reports offer less advantage than proprietary analysis.

Buying research well

Because research quality is hard to judge in advance, buyers need ways to reduce the risk of paying for poor work. Several practices help.

Specify the decision, not only the method. A brief that explains the decision the research will inform, and what would change it, allows suppliers to propose designs that fit, and allows the buyer to judge whether a proposal addresses the question.

Ask how quality will be checked. Good suppliers can explain how they will recruit respondents, check data quality, test questions and handle uncertainty. Vague answers are a warning sign.

Pay for what matters. Sample size, fieldwork quality and analytical expertise usually matter more than presentation. A polished report built on a poorly designed survey is worth less than a plain one built on a sound design.

Build relationships. Deshpandé and Zaltman (1982) found that trust between researchers and managers increased the use of research. Long-term relationships with a small number of suppliers allow them to learn the business and allow buyers to judge their quality from experience, reducing the problem of judging an experience good in advance.

Reuse and combine. Much insight can be gained by analysing existing data, combining previous studies or adding a few questions to an existing survey, at a fraction of the cost of a new study.

Measuring the return on insight

Organisations often struggle to show the return on their insight spending, because the benefit lies in decisions that were improved or mistakes that were avoided, which are hard to observe. Some evidence is available at the level of firms. Brynjolfsson et al. (2011) found that firms that reported making decisions based on data had higher output and productivity than would be expected from their other investments. Within an organisation, a practical approach is to record, for major research projects, the decision it informed, the option chosen and whether the research changed it, and to review these records periodically. Over time, this shows which kinds of research change decisions and which do not, and allows spending to shift towards the former.

The economics of experiments

Online experimentation has changed the economics of some insight. Kohavi et al. (2020) described how large online businesses run thousands of experiments at very low marginal cost, testing changes on a share of users. When experiments are cheap, it often makes sense to test rather than research in advance. But the value of experimentation also depends on the distribution of possible effects. Azevedo et al. (2020) analysed a large set of experiments at a technology company and found that the distribution of effects was "fat-tailed": most changes had small effects, but a few had very large ones. In such conditions, they showed, it can pay to run many small experiments in search of the rare big winners rather than a few large ones.

Spending wisely

Several principles follow for organisations deciding how much to spend on insight:

Key takeaways

Insight is an unusual economic good: costly to produce, cheap to copy, hard to judge before use and valuable only if it might change a decision. The value of information depends on uncertainty, stakes and the chance that research will change a decision, and can be estimated before research is commissioned. Insight is funded in several ways, from commissioned and syndicated research to public statistics, data exchange and sponsorship, each with trade-offs. Much research is wasted because organisations lack the capacity or culture to use it. Spending wisely means starting from the decision, distinguishing shared from proprietary knowledge, valuing quick research, building the capacity to use insight and tracking whether it was used.

References

Akerlof, G. A. (1970). The market for "lemons": Quality uncertainty and the market mechanism. The Quarterly Journal of Economics, 84(3), 488–500. https://doi.org/10.2307/1879431

Arrow, K. J. (1962). Economic welfare and the allocation of resources for invention. In National Bureau of Economic Research (Ed.), The rate and direction of inventive activity: Economic and social factors (pp. 609–626). Princeton University Press.

Azevedo, E. M., Deng, A., Montiel Olea, J. L., Rao, J., & Weyl, E. G. (2020). A/B testing with fat tails. Journal of Political Economy, 128(12), 4614–4672. https://doi.org/10.1086/710607

Bakos, Y., & Brynjolfsson, E. (1999). Bundling information goods: Pricing, profits, and efficiency. Management Science, 45(12), 1613–1630. https://doi.org/10.1287/mnsc.45.12.1613

Barney, J. (1991). Firm resources and sustained competitive advantage. Journal of Management, 17(1), 99–120. https://doi.org/10.1177/014920639101700108

Bergemann, D., & Bonatti, A. (2019). Markets for information: An introduction. Annual Review of Economics, 11, 85–107. https://doi.org/10.1146/annurev-economics-080315-015439

Brynjolfsson, E., Hitt, L. M., & Kim, H. H. (2011). Strength in numbers: How does data-driven decisionmaking affect firm performance? (Working paper). SSRN. https://doi.org/10.2139/ssrn.1819486

Cohen, W. M., & Levinthal, D. A. (1990). Absorptive capacity: A new perspective on learning and innovation. Administrative Science Quarterly, 35(1), 128–152. https://doi.org/10.2307/2393553

Deshpandé, R., & Zaltman, G. (1982). Factors affecting the use of market research information: A path analysis. Journal of Marketing Research, 19(1), 14–31. https://doi.org/10.1177/002224378201900102

Grossman, S. J., & Stiglitz, J. E. (1980). On the impossibility of informationally efficient markets. The American Economic Review, 70(3), 393–408.

Howard, R. A. (1966). Information value theory. IEEE Transactions on Systems Science and Cybernetics, 2(1), 22–26. https://doi.org/10.1109/TSSC.1966.300074

Hubbard, D. W. (2014). How to measure anything: Finding the value of "intangibles" in business (3rd ed.). Wiley.

Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. https://doi.org/10.1017/9781108653985

Menon, A., & Varadarajan, P. R. (1992). A model of marketing knowledge use within firms. Journal of Marketing, 56(4), 53–71. https://doi.org/10.1177/002224299205600405

Moorman, C. (1995). Organizational market information processes: Cultural antecedents and new product outcomes. Journal of Marketing Research, 32(3), 318–335. https://doi.org/10.1177/002224379503200307

Raiffa, H., & Schlaifer, R. (1961). Applied statistical decision theory. Division of Research, Graduate School of Business Administration, Harvard University.

Romer, P. M. (1990). Endogenous technological change. Journal of Political Economy, 98(5, Part 2), S71–S102. https://doi.org/10.1086/261725

Samuelson, P. A. (1954). The pure theory of public expenditure. The Review of Economics and Statistics, 36(4), 387–389. https://doi.org/10.2307/1925895

Shapiro, C., & Varian, H. R. (1998). Information rules: A strategic guide to the network economy. Harvard Business School Press.

Stigler, G. J. (1961). The economics of information. Journal of Political Economy, 69(3), 213–225. https://doi.org/10.1086/258464

Varian, H. R. (2010). Computer mediated transactions. American Economic Review, 100(2), 1–10. https://doi.org/10.1257/aer.100.2.1