Skip to content

The Generation of Knowledge · Epistemology

Knowledge From Many Voices: The Epistemology of Aggregated Opinion

When can combining many people's judgements produce knowledge that none of them holds alone? The conditions behind the "wisdom of crowds", and why independence and diversity matter more than numbers.

ModQu research article · 694 words · 6 references (APA 7th edition)

Download PDF

In 1906 the statistician Francis Galton visited a livestock fair in Plymouth, England, where visitors paid sixpence to guess the weight of an ox after it had been slaughtered and dressed. Galton collected the 787 legible tickets. The median guess was 1,207 pounds; the actual weight was 1,198 pounds, an error of less than 1% (Galton, 1907). No individual expert was needed. The crowd, taken together, knew more than almost any of its members. Galton, who had expected to show the poor judgement of ordinary people, concluded instead that the result was "more creditable to the trust-worthiness of a democratic judgment than might have been expected" (Galton, 1907, p. 451).

The episode raises a question that matters to every survey and poll: when can aggregated opinion count as knowledge?

Why averaging works

The mathematical core is old. In 1785 the Marquis de Condorcet showed that if each member of a group is more likely than not to be right on a yes-or-no question, and members judge independently, then the probability that the majority is right rises towards certainty as the group grows. Grofman et al. (1983) set out this jury theorem and its extensions, and List and Goodin (2001) generalised it to choices among more than two options.

For numerical estimates the logic is similar. Each guess combines the true value with an individual error. If errors are independent, high and low guesses cancel, and the error of the average shrinks roughly with the square root of the number of guesses. Surowiecki (2004) popularised the idea under the name "the wisdom of crowds" and identified the conditions it needs: diversity of information, independence of judgement, decentralisation, and a sound method of aggregation.

Line chart of the error of a crowd's average estimate against crowd size from 1 to 100. With independent errors, the error falls steadily towards zero. With a shared bias, the error falls at first but levels off at the size of the bias.
Figure 1. Why independence matters. When individual errors are independent, the error of the average keeps falling as the crowd grows. When everyone shares a common bias, adding more people cannot remove it. Illustrative data; chart by ModQu.

When crowds fail

The conditions are demanding, and they often fail in practice.

Shared bias. If most people err in the same direction, averaging does not cancel the error. A crowd asked to estimate the share of immigrants in its country, or the risk of a rare but vivid hazard, can be confidently and collectively wrong. As Figure 1 shows, more voices then add precision without accuracy. The same happens when everyone relies on the same source, such as one widely shared news report.

Social influence. Lorenz et al. (2011) asked groups to estimate quantities such as crime figures and then showed some participants what others had answered. Social information narrowed the range of estimates without making the average more accurate, and it made participants more confident. The crowd became less diverse and less wise at the same time.

The informed minority. Sometimes only a few people know the answer, and the majority holds a popular but wrong belief. Prelec et al. (2017) proposed a "surprisingly popular" method: ask people both for their own answer and for their prediction of how others will answer. The answer that is more popular than people predict is often correct, because those who know it also anticipate that others will not.

Lessons for survey research

A survey is a structured way of aggregating many voices, and the same conditions decide whether the result is knowledge or noise.

Aggregation does not create knowledge from nothing. It extracts the knowledge scattered across many people, provided their errors are independent enough to cancel.

References

Galton, F. (1907). Vox populi. Nature, 75(1949), 450–451. https://doi.org/10.1038/075450a0

Grofman, B., Owen, G., & Feld, S. L. (1983). Thirteen theorems in search of the truth. Theory and Decision, 15(3), 261–278. https://doi.org/10.1007/BF00125672

List, C., & Goodin, R. E. (2001). Epistemic democracy: Generalizing the Condorcet jury theorem. Journal of Political Philosophy, 9(3), 277–306. https://doi.org/10.1111/1467-9760.00128

Lorenz, J., Rauhut, H., Schweitzer, F., & Helbing, D. (2011). How social influence can undermine the wisdom of crowd effect. Proceedings of the National Academy of Sciences, 108(22), 9020–9025. https://doi.org/10.1073/pnas.1008636108

Prelec, D., Seung, H. S., & McCoy, J. (2017). A solution to the single-question crowd wisdom problem. Nature, 541(7638), 532–535. https://doi.org/10.1038/nature21054

Surowiecki, J. (2004). The wisdom of crowds. Doubleday.