Skip to content

The Generation of Knowledge · Knowledge and decisions

The Value of Information in Decision Theory

How decision theory measures the value of information, from Bayesian updating and the expected value of perfect and sample information to choosing the size of a study, with applications in health, environment and business research.

ModQu research article · 1,285 words · 13 references (APA 7th edition)

Download PDF

How much is it worth knowing something before deciding? Decision theory gives a precise answer: information is worth the expected improvement it brings to a decision. This idea, developed in the mid-twentieth century, provides a rigorous way to decide whether to collect information, how much to collect and which uncertainties matter most. This article explains the theory and its applications, with particular attention to the design of surveys and research.

Information in two senses

The word "information" has two technical meanings. Shannon (1948) defined information in terms of the reduction of uncertainty in a message, measured in bits, regardless of what the message is about or whether anyone cares. Decision theory uses a different measure. Information has value only to someone facing a choice, and only to the extent that it might change what they choose. A message that reduces uncertainty a great deal may have no decision value at all if the decision would be the same whatever it said.

Updating beliefs

Decision theory treats uncertainty in terms of probabilities that represent the decision-maker's beliefs. When new evidence arrives, beliefs are revised using Bayes' theorem, which combines the prior probability of each possibility with the likelihood of the evidence under that possibility to give a revised, or posterior, probability. Raiffa and Schlaifer (1961) developed this Bayesian approach into a systematic theory for business and economic decisions, showing how to evaluate experiments and samples before they are carried out.

The key step is to look ahead. Before collecting information, the decision-maker considers each result the study might produce, how likely each result is and what they would decide after seeing it. Averaging over the possible results gives the expected value of deciding with the information. Comparing this with the expected value of deciding now gives the value of the information (Pratt et al., 1995).

Perfect and sample information

Howard (1966) set out information value theory and distinguished two quantities.

The expected value of perfect information is the expected gain from learning the true state of the world before deciding. It is calculated by imagining that the uncertainty is resolved, finding the best decision in each case and comparing the average outcome with that of the best decision under current uncertainty. Because no real study gives perfect information, this value sets an upper limit on what any research could be worth. If it is small, no research on that question is worth much.

The expected value of sample information is the expected gain from a particular, imperfect study. It is always less than the value of perfect information, and it depends on the accuracy of the study: on its size, its design and its freedom from bias.

Two further results follow. First, information can never have negative expected value to a rational decision-maker, because they can always ignore it. Second, information has zero value if no possible result would change the decision. Blackwell (1953) showed more generally how to compare experiments: one experiment is more informative than another if every decision-maker, whatever their goals, would prefer it.

How large should a study be?

The value of sample information grows with the size of the sample, but at a diminishing rate, approaching the value of perfect information as a ceiling. The cost of a study, on the other hand, usually rises steadily with its size. The best size is where the value of information minus its cost is greatest. Figure 1 illustrates this.

Illustrative chart of the value and cost of a survey by sample size: the value of the survey rises steeply and levels off below the value of perfect information, cost rises in a straight line, and net value peaks at a sample of about 800.
Figure 1. Choosing the size of a study by the value of information: an illustrative example. Method after Raiffa and Schlaifer (1961); calculation and chart by ModQu.

This approach differs from the usual way of choosing sample sizes, which aims for a given level of statistical precision or power. Claxton (1999) argued in the context of health care that conventional rules for statistical significance are irrelevant to decisions: what matters is the expected consequence of choosing one option rather than another, and whether further research is worth its cost. A study that is too small to reach statistical significance may still be worth doing if it could change a decision, and a large study may be wasteful if the decision is already clear.

Which uncertainties matter?

Decision problems usually involve many uncertain quantities. Value-of-information analysis can identify which of them is worth reducing. Felli and Hazen (1998) showed that the expected value of perfect information about each quantity is a better guide to its importance than conventional sensitivity analysis, because it accounts for both how uncertain the quantity is and how much the decision depends on it.

Hubbard (2014) applied this reasoning to business decisions and found that organisations often measure the quantities that are easiest to measure rather than those with the highest information value. He argued that a few well-chosen measurements, directed at the most decision-relevant uncertainties, are usually worth far more than extensive measurement of everything.

Applications

Value-of-information analysis is now used in several fields. In health care, it helps decide whether a new treatment should be adopted now or whether further trials are needed first, and which trials would be most valuable (Claxton, 1999; Wilson, 2015). In environmental management, Keisler et al. (2014) reviewed a growing body of applications, in which the method was used to decide what to monitor, how to prioritise research and when to act rather than study further. They noted that while the theory was well established, practical use was held back by the effort needed to build decision models and the difficulty of expressing uncertainty as probabilities.

In business and market research, the same logic applies to decisions about launching products, entering markets and changing prices. Raiffa (1968) presented many such examples in an introduction to decision analysis written for managers.

Limits of the approach

Value-of-information analysis rests on assumptions that are not always met. It requires the decision-maker to express beliefs as probabilities, which Savage (1954) showed can be done consistently in principle but which many people find difficult in practice, especially for unfamiliar events. It requires a model linking uncertain quantities to outcomes, and the results are only as good as that model. And it assumes that the possible states of the world can be listed in advance. Kay and King (2020) argued that many important decisions involve radical uncertainty, in which the possibilities themselves are unclear. In such cases, the formal calculation may give false precision, though its central question, whether a piece of information could change what we would do, remains useful.

Implications for survey research

Value-of-information thinking offers survey researchers and their clients a disciplined way to plan research.

Start from the decision. Identify the choice the research will inform and the outcomes that would lead to different choices.

Find the critical uncertainties. Concentrate effort on the quantities that most affect the decision, not on everything that could be measured.

Size the study to the decision. Choose a sample large enough to change the decision when it should, but not so large that the extra precision is worth less than its cost.

Account for quality. Bias reduces the value of information as surely as small samples do, and is not reduced by larger samples. Investing in sound sampling and measurement can add more value than adding interviews.

Conclusion

Decision theory values information by its expected effect on decisions. The expected value of perfect information sets a ceiling on what research can be worth; the expected value of sample information measures what a particular study adds. These ideas show when research is worthwhile, how large a study should be and which uncertainties deserve attention. Although building the necessary models takes effort, the reasoning behind them can guide everyday choices about research: information is valuable when it can change what we do.

References

Blackwell, D. (1953). Equivalent comparisons of experiments. Annals of Mathematical Statistics, 24(2), 265–272. https://doi.org/10.1214/aoms/1177729032

Claxton, K. (1999). The irrelevance of inference: A decision-making approach to the stochastic evaluation of health care technologies. Journal of Health Economics, 18(3), 341–364. https://doi.org/10.1016/S0167-6296(98)00039-3

Felli, J. C., & Hazen, G. B. (1998). Sensitivity analysis and the expected value of perfect information. Medical Decision Making, 18(1), 95–109. https://doi.org/10.1177/0272989X9801800117

Howard, R. A. (1966). Information value theory. IEEE Transactions on Systems Science and Cybernetics, 2(1), 22–26. https://doi.org/10.1109/TSSC.1966.300074

Hubbard, D. W. (2014). How to measure anything: Finding the value of "intangibles" in business (3rd ed.). Wiley.

Kay, J., & King, M. (2020). Radical uncertainty: Decision-making beyond the numbers. W. W. Norton.

Keisler, J. M., Collier, Z. A., Chu, E., Sinatra, N., & Linkov, I. (2014). Value of information analysis: The state of application. Environment Systems and Decisions, 34(1), 3–23. https://doi.org/10.1007/s10669-013-9439-4

Pratt, J. W., Raiffa, H., & Schlaifer, R. (1995). Introduction to statistical decision theory. MIT Press.

Raiffa, H. (1968). Decision analysis: Introductory lectures on choices under uncertainty. Addison-Wesley.

Raiffa, H., & Schlaifer, R. (1961). Applied statistical decision theory. Division of Research, Graduate School of Business Administration, Harvard University.

Savage, L. J. (1954). The foundations of statistics. Wiley.

Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x

Wilson, E. C. F. (2015). A practical guide to value of information analysis. PharmacoEconomics, 33(2), 105–121. https://doi.org/10.1007/s40273-014-0219-x