Skip to content

Intelligence · Artificial and collective intelligence

Surveys as Instruments of Collective Intelligence

A well-designed survey is one of the most effective ways to gather and combine the knowledge of many people. This article explains how surveys put the principles of collective intelligence into practice, the different kinds of knowledge they can capture, and the design choices that keep the result wise.

ModQu research article · 1,680 words · 17 references (APA 7th edition)

Download PDF

Collective intelligence is often described with examples such as prediction markets, crowdsourcing platforms and online communities. Yet the oldest and most widely used instrument for combining the knowledge of many people is far more familiar: the survey. A survey asks a defined group of people the same questions, separately, and combines their answers by a fixed rule. In doing so, it meets almost exactly the conditions that research has identified for a wise crowd. Understanding surveys in this way helps explain both why they work and why they sometimes fail.

The survey as a wise crowd

Surowiecki (2004) set out four conditions for collective wisdom: diversity of opinion, independence of judgement, decentralised knowledge and a method of aggregation. A well-designed survey builds each one into its method.

Diversity comes from sampling. A survey that reaches a broad cross-section of the relevant population captures the range of experience and information held across it. Neyman (1934) and later Kish (1965) established the statistical foundations for selecting samples that represent a population, so that the variety of views in the sample reflects the variety in the population.

Independence comes from how questions are asked. Each respondent answers privately, without seeing others' responses. This protects against the social influence that Lorenz et al. (2011) showed can make crowds more confident without making them more accurate.

Decentralisation is inherent: each respondent reports from their own position, whether as a customer, an employee, a citizen or a professional. Hayek (1945) argued that knowledge of particular circumstances of time and place is dispersed among many people and cannot be gathered by any central authority. A survey is one of the few practical means of collecting it.

Aggregation is the final step. Answers are combined into percentages, averages or distributions by rules decided in advance. Figure 1 shows how these steps fit together.

Five-step process: define the population, sample widely, ask independently, weight to the population, and aggregate by a fixed rule.
Figure 1. From individual answers to collective knowledge. Each stage of a survey protects one of the conditions for a wise crowd. Based on Groves et al. (2009) and Surowiecki (2004); diagram by ModQu.

A short history

Converse (1987) traces the development of survey research in the United States from social surveys of poverty in the late nineteenth century to the scientific polling and academic survey centres of the mid-twentieth. Groves (2011) describes three eras: an era of invention, from the 1930s to the 1960s, when the basic methods of sampling and questioning were developed; an era of expansion, when telephone surveys made data collection faster and cheaper; and the current era, marked by falling response rates, online data collection and the availability of other kinds of data. Throughout, the core idea has remained the same: asking many people, carefully chosen, the same well-designed questions produces knowledge that no single observer could provide.

Three kinds of collective knowledge

Surveys can capture different kinds of knowledge, depending on what respondents are asked about. Figure 2 distinguishes three.

Table of three kinds of survey question: about the respondent, about expectations, and about other people, with example, what is aggregated and what it reveals.
Figure 2. Three kinds of survey question, three kinds of collective knowledge. Compiled from Groves et al. (2009), Graefe (2014) and Prelec et al. (2017); table by ModQu.

Questions about the respondent

Most survey questions ask people about themselves: their behaviour, experiences, attitudes and intentions. Aggregated, the answers describe the state of a population, such as the share of households that own a pet, the level of satisfaction with a service or the proportion of voters who support a policy. This is collective intelligence in a straightforward sense: knowledge that exists only in scattered form, one person at a time, combined into a picture of the whole.

Questions about expectations

A second kind of question asks people what they expect to happen. Here the survey works like Galton's (1907) ox-weighing crowd: each respondent makes a judgement using their own information, and the aggregate is a forecast. Graefe (2014) compared the accuracy of methods used to forecast American presidential elections and found that vote expectation surveys, which ask people who they think will win rather than how they will vote, were more accurate than intention polls, expert judgement and several statistical models. Respondents draw on what they hear from family, friends and colleagues, so each answer carries information about many people, not one.

Questions about other people

A third kind asks respondents what they think other people believe or do. Prelec, Seung and McCoy (2017) showed that combining answers to "what do you think?" with answers to "what do you think others will say?" can reveal when a knowledgeable minority holds the correct view. An answer that is more common than respondents predicted is often the right one. Questions about others can also reduce the pressure to give socially desirable answers about oneself.

Designing for wise answers

The quality of the collective answer depends on the quality of each individual answer. Tourangeau, Rips and Rasinski (2000) describe answering a survey question as a sequence of tasks: understanding the question, retrieving relevant memories, forming a judgement and reporting it in the format offered. Error can enter at each stage. Krosnick (1999) showed that when questions are difficult or respondents are unmotivated, people often take mental shortcuts, choosing the first reasonable option or agreeing with whatever is proposed. These shortcuts add shared, systematic errors that do not cancel out in aggregation.

Groves et al. (2009) organise survey error into two broad families. Errors of representation arise when the people who answer differ from the population of interest, through gaps in the list from which the sample is drawn, through chance variation in sampling, or through differences between those who respond and those who do not. Errors of measurement arise when answers differ from the truth, through question wording, the way the survey is administered, or respondents' memory and motivation. Collective intelligence is undermined by either kind, but especially by errors that push many respondents in the same direction.

Dillman, Smyth and Christian (2014) set out practical methods for reducing both: clear and neutral question wording, logical ordering, careful visual design for online questionnaires, and contact strategies that encourage a broad range of people to respond. Couper (2017) reviews newer developments, including mobile data collection and combinations of survey data with other sources, and their implications for data quality.

When the crowd misleads

Surveys share the failure modes of other crowds. If the sample leaves out part of the population, the result reflects only those included. Kennedy et al. (2018), in an evaluation of polling in the 2016 American presidential election, found that national polls were reasonably accurate but many state polls underestimated support for the winning candidate. One important reason was that many polls did not adjust their samples for education, at a time when education had become strongly linked to voting choice. The crowd was asked, but not the right crowd in the right proportions.

Surveys can also lose independence. Showing respondents the results so far, or framing a question around what "most people" think, introduces the social influence that narrows answers. Leading wording pushes respondents in a common direction, creating exactly the kind of shared bias that averaging cannot remove.

Surveys and deliberation

A survey captures what people think now, often without much reflection. Fishkin (2009) developed deliberative polling to capture what people would think if they had more information and time to consider it. A representative sample is surveyed, then brought together to receive balanced briefing materials and discuss the issues in small groups, and surveyed again. Opinions often change substantially. The method combines the representativeness of a survey with the informed judgement of a deliberating group, while protecting independence through private questionnaires before and after.

An illustration

A home furnishings retailer is deciding which of three new sofa designs to stock in volume. A conventional approach would ask a sample of customers which design they prefer. The intelligence team adds two further questions. It asks each respondent which design they think will be most popular among people they know, and what share of their friends they expect to choose each one. It also runs a short internal survey of store staff, who see thousands of customers each month, asking which design they expect to sell best.

The results differ in instructive ways. Customers' own preferences are split almost evenly between two designs. Their expectations about others, however, favour one design clearly, and more respondents chose it as their own preference than the group had predicted. Store staff, answering separately, rank the same design first. Taken together, the three kinds of question point to a clear choice that the preference question alone did not reveal. The retailer stocks that design in volume and the other two in smaller quantities. Each survey asked a different crowd a different question, and the combination proved more informative than any single measure.

Applying the idea in business

For organisations, treating surveys as instruments of collective intelligence suggests several practices:

  1. Ask expectation questions as well as preference questions. "Which of these products do you think will sell best?" often adds information that "Which would you buy?" misses.
  2. Survey internal experts. Sales staff, engineers and customer service teams hold dispersed knowledge about markets. A structured internal survey collects it without the anchoring of a meeting.
  3. Protect independence. Do not show running results or describe what others have said.
  4. Weight to the population. Compare the sample with known population figures and adjust where groups are under- or over-represented.
  5. Look at the spread, not only the average. Wide disagreement is information; it signals uncertainty or distinct groups.
  6. Track accuracy. Where survey answers predict outcomes, compare them with what happened and refine methods over time.

Conclusion

Surveys are among the oldest and most reliable tools of collective intelligence. By sampling widely, asking independently and combining answers by fixed rules, they turn knowledge scattered across many people into a single, usable picture. Their accuracy depends on the same conditions as any wise crowd: a diverse and representative group, freedom from shared bias, and sound aggregation. Designed with these conditions in mind, a survey is not just a measurement of opinion but a way of making a group smarter than any of its members.

References

Converse, J. M. (1987). Survey research in the United States: Roots and emergence, 1890–1960. University of California Press.

Couper, M. P. (2017). New developments in survey data collection. Annual Review of Sociology, 43, 121–145. https://doi.org/10.1146/annurev-soc-060116-053613

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Wiley. https://doi.org/10.1002/9781394260645

Fishkin, J. S. (2009). When the people speak: Deliberative democracy and public consultation. Oxford University Press.

Galton, F. (1907). Vox populi. Nature, 75(1949), 450–451. https://doi.org/10.1038/075450a0

Graefe, A. (2014). Accuracy of vote expectation surveys in forecasting elections. Public Opinion Quarterly, 78(S1), 204–232. https://doi.org/10.1093/poq/nfu008

Groves, R. M. (2011). Three eras of survey research. Public Opinion Quarterly, 75(5), 861–871. https://doi.org/10.1093/poq/nfr057

Groves, R. M., Fowler, F. J., Jr., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology (2nd ed.). Wiley.

Hayek, F. A. (1945). The use of knowledge in society. American Economic Review, 35(4), 519–530.

Kennedy, C., Blumenthal, M., Clement, S., Clinton, J. D., Durand, C., Franklin, C., McGeeney, K., Miringoff, L., Olson, K., Rivers, D., Saad, L., Witt, G. E., & Wlezien, C. (2018). An evaluation of the 2016 election polls in the United States. Public Opinion Quarterly, 82(1), 1–33. https://doi.org/10.1093/poq/nfx047

Kish, L. (1965). Survey sampling. Wiley.

Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537–567. https://doi.org/10.1146/annurev.psych.50.1.537

Lorenz, J., Rauhut, H., Schweitzer, F., & Helbing, D. (2011). How social influence can undermine the wisdom of crowd effect. Proceedings of the National Academy of Sciences, 108(22), 9020–9025. https://doi.org/10.1073/pnas.1008636108

Neyman, J. (1934). On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection. Journal of the Royal Statistical Society, 97(4), 558–625. https://doi.org/10.2307/2342192

Prelec, D., Seung, H. S., & McCoy, J. (2017). A solution to the single-question crowd wisdom problem. Nature, 541(7638), 532–535. https://doi.org/10.1038/nature21054

Surowiecki, J. (2004). The wisdom of crowds. Doubleday.

Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press. https://doi.org/10.1017/CBO9780511819322