The Generation of Knowledge · Epistemology
Knowledge From Many Voices: The Epistemology of Aggregated Opinion
When can combining many people's judgements produce knowledge that none of them holds alone? The conditions behind the "wisdom of crowds", and why independence and diversity matter more than numbers.
In 1906 the statistician Francis Galton visited a livestock fair in Plymouth, England, where visitors paid sixpence to guess the weight of an ox after it had been slaughtered and dressed. Galton collected the 787 legible tickets. The median guess was 1,207 pounds; the actual weight was 1,198 pounds, an error of less than 1% (Galton, 1907). No individual expert was needed. The crowd, taken together, knew more than almost any of its members. Galton, who had expected to show the poor judgement of ordinary people, concluded instead that the result was "more creditable to the trust-worthiness of a democratic judgment than might have been expected" (Galton, 1907, p. 451).
The episode raises a question that matters to every survey and poll: when can aggregated opinion count as knowledge?
Why averaging works
The mathematical core is old. In 1785 the Marquis de Condorcet showed that if each member of a group is more likely than not to be right on a yes-or-no question, and members judge independently, then the probability that the majority is right rises towards certainty as the group grows. Grofman et al. (1983) set out this jury theorem and its extensions, and List and Goodin (2001) generalised it to choices among more than two options.
For numerical estimates the logic is similar. Each guess combines the true value with an individual error. If errors are independent, high and low guesses cancel, and the error of the average shrinks roughly with the square root of the number of guesses. Surowiecki (2004) popularised the idea under the name "the wisdom of crowds" and identified the conditions it needs: diversity of information, independence of judgement, decentralisation, and a sound method of aggregation.
When crowds fail
The conditions are demanding, and they often fail in practice.
Shared bias. If most people err in the same direction, averaging does not cancel the error. A crowd asked to estimate the share of immigrants in its country, or the risk of a rare but vivid hazard, can be confidently and collectively wrong. As Figure 1 shows, more voices then add precision without accuracy. The same happens when everyone relies on the same source, such as one widely shared news report.
Social influence. Lorenz et al. (2011) asked groups to estimate quantities such as crime figures and then showed some participants what others had answered. Social information narrowed the range of estimates without making the average more accurate, and it made participants more confident. The crowd became less diverse and less wise at the same time.
The informed minority. Sometimes only a few people know the answer, and the majority holds a popular but wrong belief. Prelec et al. (2017) proposed a "surprisingly popular" method: ask people both for their own answer and for their prediction of how others will answer. The answer that is more popular than people predict is often correct, because those who know it also anticipate that others will not.
Lessons for survey research
A survey is a structured way of aggregating many voices, and the same conditions decide whether the result is knowledge or noise.
- Know what kind of question you are asking. A survey of attitudes measures what people think; that is knowledge about the population. A survey that asks people to estimate facts is aggregating judgements, and is only as good as the independence and information of the respondents.
- Protect independence. Showing respondents previous results, or asking leading questions, can create the herding that Lorenz et al. (2011) observed.
- Seek diversity, not just size. A large sample from one narrow group shares that group's biases. A well-designed sample covers the range of experience in the population.
- Look beyond the majority. Asking respondents what they expect others to think, as well as what they think themselves, can reveal where real knowledge sits.
Aggregation does not create knowledge from nothing. It extracts the knowledge scattered across many people, provided their errors are independent enough to cancel.
References
Galton, F. (1907). Vox populi. Nature, 75(1949), 450–451. https://doi.org/10.1038/075450a0
Grofman, B., Owen, G., & Feld, S. L. (1983). Thirteen theorems in search of the truth. Theory and Decision, 15(3), 261–278. https://doi.org/10.1007/BF00125672
List, C., & Goodin, R. E. (2001). Epistemic democracy: Generalizing the Condorcet jury theorem. Journal of Political Philosophy, 9(3), 277–306. https://doi.org/10.1111/1467-9760.00128
Lorenz, J., Rauhut, H., Schweitzer, F., & Helbing, D. (2011). How social influence can undermine the wisdom of crowd effect. Proceedings of the National Academy of Sciences, 108(22), 9020–9025. https://doi.org/10.1073/pnas.1008636108
Prelec, D., Seung, H. S., & McCoy, J. (2017). A solution to the single-question crowd wisdom problem. Nature, 541(7638), 532–535. https://doi.org/10.1038/nature21054
Surowiecki, J. (2004). The wisdom of crowds. Doubleday.