Surveys · Sampling
Recruiting Survey Respondents Through Search: Reach, Relevance and Bias
Search engines can bring survey invitations to people at the moment they are looking for information on a topic. This article explains how search-based recruitment works, what it does well, the biases it can introduce and how researchers can measure and reduce them.
Every day, people type billions of questions into search engines. Someone searching for information about home insurance, a medical symptom or the best laptop for students is telling the world, at that moment, what they are interested in. For survey researchers, this creates an opportunity: a survey invitation shown to people searching a topic reaches respondents who are, by definition, engaged with that topic. Recruitment through search, whether through paid search advertisements, survey pages that rank in organic search results or invitations placed on pages that searchers visit, has become one of several online routes to finding respondents (Lane et al., 2015).
The opportunity comes with a challenge. People who search for a topic, see an invitation and choose to take part are not a random selection of the population. This article explains how search-based recruitment works, where it is useful, how it can bias results and what researchers can do about it. It discusses search recruitment in general terms; specific organisations use their own methods.
How search recruitment works
There are three main routes.
Paid search advertising. Researchers buy advertisements that appear alongside search results for chosen keywords. A study on smoking might advertise against searches for "quit smoking"; a consumer study might use product-related terms. People who click are taken to a landing page that explains the study and leads to the survey. Paid search allows precise targeting by keyword, location and sometimes time of day (Antoun et al., 2016).
Organic search. Pages that host or describe a survey, or that publish content on the topic with an invitation to take part, can appear in the ordinary search results. People who find these pages while searching may follow the invitation. Organic traffic costs little per respondent once the pages exist, but it depends on how search engines rank the pages, which the researcher does not control.
Search-referred website traffic. Many people arrive at websites through search, and an invitation on such a site reaches them indirectly. This route overlaps with river sampling and website intercept surveys (Couper, 2000).
What search recruitment does well
Reach
Search engines are used by a very large share of internet users, so search-based recruitment can reach people who would never join a survey panel or respond to an email invitation. Health researchers have used online advertising to recruit hard-to-reach groups quickly and at low cost, including young adults and people with specific conditions (Ramo & Prochaska, 2012; Whitaker et al., 2017).
Relevance
Because invitations are tied to what people are searching for, search recruitment can find people with a specific interest, need or recent experience. For studies about a particular product category, health condition or life event, this can produce a sample with far more relevant respondents than a general population sample of the same size. It can also reach people close to the moment of a decision, when their experiences and intentions are fresh.
Speed and cost
Paid search campaigns can be started and stopped within hours, and costs can be controlled by bidding limits. A comparison of online recruitment channels found that cost per completed survey varied widely between channels, with search advertising neither the cheapest nor the most expensive (Antoun et al., 2016).
Where the bias comes from
Figure 1 shows four filters that stand between a population and a sample recruited through search. Each can make the sample differ from the population.
Who searches
Search recruitment can reach only people who are online and who use search engines. Internet access and the skills to use it vary with age, education and income, even among young people who are often assumed to be uniformly skilled (Hargittai, 2010). People who search frequently and in sophisticated ways are more likely to be reached than those who rarely search. This is a form of coverage error.
What they search for
The people searching for a topic are, almost by definition, more interested in it than the population as a whole. Research on survey participation shows that interest in the topic is one of the strongest influences on whether people take part, and that it can produce nonresponse bias in estimates related to that topic (Groves et al., 2004; Groves et al., 2006). Leverage-saliency theory explains why: a survey whose topic matters to someone is more salient, and they are more likely to say yes (Groves et al., 2000). In search recruitment, this effect is built into the recruitment channel itself. A survey about exercise recruited through searches for "running shoes" will over-represent active people.
Search behaviour itself also changes over time, in response to news, seasons and the search engine's own suggestions. Analyses of search data for tracking illness showed how changes in search behaviour, unrelated to the underlying phenomenon, can mislead (Ginsberg et al., 2009; Lazer et al., 2014). The same instability affects who is recruited through search from one week to the next.
What they see
Not every searcher sees an invitation. In organic search, most users click on results near the top of the first page, and studies of user behaviour show that the position of a result strongly affects whether it is clicked (Pan et al., 2007). In paid search, the advertising platform's algorithms decide which users see which advertisements, often optimising for clicks rather than for representativeness. The researcher may not know the rules, and the platform's choices can change without notice.
Who takes part
Finally, among those who see the invitation, only a small fraction click, and fewer complete the survey. Figure 2 gives an illustrative funnel. At each step, the decision to continue may be related to the person's views or experiences. People with strong opinions, recent problems or time to spare are more likely to take part (Bethlehem, 2010).
Measuring the bias
Because search-recruited samples are nonprobability samples, their accuracy cannot be guaranteed by design. It can, however, be assessed (Baker et al., 2013).
- Benchmark questions. Include questions whose population values are known from official statistics or high-quality surveys, such as household composition, employment or general internet use, and compare.
- Source tracking. Record the keyword, advertisement, page or referral source for each respondent, and compare answers across sources. Large differences suggest that the source is shaping results.
- Comparison with other samples. Where budget allows, run the same core questions on a probability sample or a different nonprobability source and compare.
- Paradata. Time of day, device and time spent on the landing page can reveal differences in how respondents arrived and engaged (Callegaro et al., 2014).
Reducing the bias
Diversify the routes
Using many keywords, including general as well as specific terms, broadens the range of people reached. Combining search with other recruitment routes, such as social media, panels and email, reduces dependence on one channel's biases (Lane et al., 2015).
Use quotas and weights
Quotas during recruitment and weighting afterwards can align the sample with the population on known characteristics. Weighting works best when it includes variables related both to the chance of being recruited and to the survey topic, such as internet use and interest in the topic, not only demographics (Mercer et al., 2017; Elliott & Valliant, 2017).
Define the population to fit the method
Sometimes the most honest approach is to define the target population as the people the method reaches. A survey of "people who searched for information about home renovation in the past month" may be exactly what a business needs, and search recruitment is well suited to it. Problems arise when results from such a sample are presented as describing all homeowners.
Control quality
Online recruitment attracts some respondents who are motivated only by rewards, as well as automated programs. Attention checks, consistency checks, limits on repeat participation and screening of implausible answers are necessary (Callegaro et al., 2014). Reporting standards for internet surveys recommend describing recruitment, incentives and checks in detail (Eysenbach, 2004).
How search compares with other online routes
Search is one of several ways to recruit respondents online. Online panels draw on a standing pool of members who have agreed to take surveys. Social media advertising targets people by their profiles and interests rather than by what they are looking for at the moment. Email invitations reach people on a list. River and intercept methods catch people on websites. Each route reaches a different slice of the online population and introduces its own selection effects (Callegaro et al., 2014).
Early reviews of internet research noted that the main attractions of web recruitment, speed and low cost, came with uncertain coverage and self-selection, and that results depended heavily on how respondents were found (Fricker & Schonlau, 2002). Direct comparisons bear this out. When the same survey was promoted through several channels, the demographic profile, data quality and cost of the resulting samples differed from channel to channel (Antoun et al., 2016). The distinctive feature of search is its link to intent: it recruits people while they are actively looking for something. This makes it strong for reaching people engaged with a topic and weak for representing those who are not.
Methodologists increasingly see such sources as complements to designed samples rather than substitutes. One proposal is to use a smaller probability sample to calibrate or check estimates from larger, cheaper nonprobability sources, combining the accuracy of the first with the size and speed of the second (Brick, 2011). A search-recruited sample can play this role well when key variables are measured in both.
Landing pages and questionnaire design
The page that a searcher reaches after clicking has a strong influence on who continues. A landing page that promises a reward prominently attracts reward-seekers; one that stresses the topic attracts the most interested. Neutral wording that explains the purpose, the time needed and how data will be used tends to produce a broader mix of respondents. Because many searchers arrive on phones, the landing page and questionnaire must work well on small screens, and the first questions should be quick and easy to answer, since people who arrive from search have not committed in advance to taking a survey.
Reporting search-recruited studies
Reports should state that respondents were recruited through search, which search engines and keywords were used, the dates of recruitment, any incentives, the number of people who clicked, started and completed, and the checks applied to remove poor-quality responses. They should describe the population the sample represents and the adjustments made, and should not present margins of error that assume random selection (Eysenbach, 2004; Baker et al., 2013).
When search recruitment fits
Search recruitment is best suited to studies where:
- The target population is defined by an interest, need or recent experience that people search for.
- Comparisons within the sample, such as between product features or groups of users, matter more than population totals.
- Speed and the ability to reach people near a decision are important.
- The researcher can track sources, include benchmarks and adjust the results.
It is least suited to estimating the prevalence of a behaviour or opinion in the general population, especially when that behaviour is related to searching itself.
An example
A financial services firm wants to understand what people look for when choosing a first mortgage. It runs search advertisements against terms such as "first-time buyer mortgage" and "how much can I borrow", and recruits 1,500 respondents in three weeks. The sample is relevant: most respondents are actively considering a mortgage, which a general population sample would contain few of. The firm records the keyword that brought each respondent, finds that people who searched for borrowing limits are younger and less confident than those who searched for interest rates, and reports results by search route. It does not use the sample to estimate how many adults plan to buy a home, because people who are not yet searching are missing (illustrative example).
Ethical considerations
Search-based recruitment uses advertising systems that collect data about users. Researchers should ensure that landing pages explain the study clearly, state who is conducting it, describe how data will be used and obtain informed consent before the survey begins. They should avoid targeting that could be intrusive, for example advertising against searches for sensitive health conditions without care for how the invitation might be perceived (Whitaker et al., 2017). Incentives should be described honestly, and personal data collected through advertising platforms should be handled under the relevant privacy rules.
Conclusion
Search engines give survey researchers a way to reach people at the moment they are interested in a topic. This provides reach, relevance and speed that few other methods can match. The same features create bias: respondents are people who search, who searched for this topic, who were shown the invitation and who chose to take part. These filters can be measured with benchmarks and source tracking, reduced with diverse recruitment and careful weighting, and made transparent by defining the population honestly. Used with this care, search recruitment can generate useful knowledge, especially about people engaged with a topic. Used without it, large samples can give confident but misleading answers (Meng, 2018; Cornesse et al., 2020).
References
Antoun, C., Zhang, C., Conrad, F. G., & Schober, M. F. (2016). Comparisons of online recruitment strategies for convenience samples: Craigslist, Google AdWords, Facebook, and Amazon Mechanical Turk. Field Methods, 28(3), 231–246. https://doi.org/10.1177/1525822X15603149
Baker, R., Brick, J. M., Bates, N. A., Battaglia, M., Couper, M. P., Dever, J. A., Gile, K. J., & Tourangeau, R. (2013). Summary report of the AAPOR task force on non-probability sampling. Journal of Survey Statistics and Methodology, 1(2), 90–143. https://doi.org/10.1093/jssam/smt008
Bethlehem, J. (2010). Selection bias in web surveys. International Statistical Review, 78(2), 161–188. https://doi.org/10.1111/j.1751-5823.2010.00112.x
Brick, J. M. (2011). The future of survey sampling. Public Opinion Quarterly, 75(5), 872–888. https://doi.org/10.1093/poq/nfr045
Callegaro, M., Baker, R., Bethlehem, J., Göritz, A. S., Krosnick, J. A., & Lavrakas, P. J. (Eds.). (2014). Online panel research: A data quality perspective. Wiley. https://doi.org/10.1002/9781118763520
Cornesse, C., Blom, A. G., Dutwin, D., Krosnick, J. A., De Leeuw, E. D., Legleye, S., Pasek, J., Pennay, D., Phillips, B., Sakshaug, J. W., Struminskaya, B., & Wenz, A. (2020). A review of conceptual approaches and empirical evidence on probability and nonprobability sample survey research. Journal of Survey Statistics and Methodology, 8(1), 4–36. https://doi.org/10.1093/jssam/smz041
Couper, M. P. (2000). Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64(4), 464–494. https://doi.org/10.1086/318641
Elliott, M. R., & Valliant, R. (2017). Inference for nonprobability samples. Statistical Science, 32(2), 249–264. https://doi.org/10.1214/16-STS598
Eysenbach, G. (2004). Improving the quality of Web surveys: The Checklist for Reporting Results of Internet E-Surveys (CHERRIES). Journal of Medical Internet Research, 6(3), e34. https://doi.org/10.2196/jmir.6.3.e34
Fricker, R. D., Jr., & Schonlau, M. (2002). Advantages and disadvantages of Internet research surveys: Evidence from the literature. Field Methods, 14(4), 347–367. https://doi.org/10.1177/152582202237725
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009). Detecting influenza epidemics using search engine query data. Nature, 457(7232), 1012–1014. https://doi.org/10.1038/nature07634
Groves, R. M., Couper, M. P., Presser, S., Singer, E., Tourangeau, R., Acosta, G. P., & Nelson, L. (2006). Experiments in producing nonresponse bias. Public Opinion Quarterly, 70(5), 720–736. https://doi.org/10.1093/poq/nfl036
Groves, R. M., Presser, S., & Dipko, S. (2004). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68(1), 2–31. https://doi.org/10.1093/poq/nfh002
Groves, R. M., Singer, E., & Corning, A. (2000). Leverage-saliency theory of survey participation: Description and an illustration. Public Opinion Quarterly, 64(3), 299–308. https://doi.org/10.1086/317990
Hargittai, E. (2010). Digital na(t)ives? Variation in internet skills and uses among members of the "net generation". Sociological Inquiry, 80(1), 92–113. https://doi.org/10.1111/j.1475-682X.2009.00317.x
Lane, T. S., Armin, J., & Gordon, J. S. (2015). Online recruitment methods for web-based and mobile health studies: A review of the literature. Journal of Medical Internet Research, 17(7), e183. https://doi.org/10.2196/jmir.4359
Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014). The parable of Google Flu: Traps in big data analysis. Science, 343(6176), 1203–1205. https://doi.org/10.1126/science.1248506
Meng, X.-L. (2018). Statistical paradises and paradoxes in big data (I): Law of large populations, big data paradox, and the 2016 US presidential election. The Annals of Applied Statistics, 12(2), 685–726. https://doi.org/10.1214/18-AOAS1161SF
Mercer, A. W., Kreuter, F., Keeter, S., & Stuart, E. A. (2017). Theory and practice in nonprobability surveys: Parallels between causal inference and survey inference. Public Opinion Quarterly, 81(S1), 250–271. https://doi.org/10.1093/poq/nfw060
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007). In Google we trust: Users' decisions on rank, position, and relevance. Journal of Computer-Mediated Communication, 12(3), 801–823. https://doi.org/10.1111/j.1083-6101.2007.00351.x
Ramo, D. E., & Prochaska, J. J. (2012). Broad reach and targeted recruitment using Facebook for an online survey of young adult substance use. Journal of Medical Internet Research, 14(1), e28. https://doi.org/10.2196/jmir.1878
Whitaker, C., Stevelink, S., & Fear, N. (2017). The use of Facebook in recruiting participants for health research purposes: A systematic review. Journal of Medical Internet Research, 19(8), e290. https://doi.org/10.2196/jmir.7071