Surveys · Sampling
Topic-Targeted Recruitment: Matching Respondents to the Questions Asked
Topic-targeted recruitment seeks respondents who are relevant to a survey's subject, such as users of a product or people with a particular experience. This article explains why it is attractive, how it changes who takes part, and how to keep the resulting knowledge sound.
A survey about electric vehicles is of little use if most respondents have never thought about buying one. A study of the experience of hospital discharge needs people who have recently left hospital. For many questions, the people who matter are a small part of the population, and a general sample would contain few of them. Topic-targeted recruitment tries to solve this by seeking out respondents who are relevant to the questions being asked: people who use a product, have an interest, share an experience or face a decision.
Targeting can make surveys more efficient and their answers more informed. It also changes the composition of the sample in ways that can mislead if they are not understood. This article explains the forms targeting takes, what it does to a sample and how to use it well.
Forms of topic targeting
Targeting can occur at several points in recruitment.
- Screening a general sample. Many people are contacted, and a few short questions identify those who are eligible. This is the classic approach for rare groups in probability surveys (Kalton & Anderson, 1986).
- Recruiting from topic-related places. Invitations are placed where relevant people are likely to be: specialist websites, forums, product pages, clinics or events.
- Targeted online advertising. Advertising platforms allow invitations to be shown to people based on their search terms, interests or behaviour (Ramo & Prochaska, 2012; Antoun et al., 2016).
- Panel profiling. Online panels store information about their members, such as product ownership or health conditions, and invite only those who match (Callegaro et al., 2014).
- Topic-led invitations. The invitation itself names the subject of the survey, which encourages interested people to take part.
Why targeting is attractive
Efficiency
If 5 per cent of adults own a product, a general sample of 2,000 would include about 100 owners. A targeted sample can deliver 1,000 owners for a similar cost. When the questions are only meaningful for a specific group, this efficiency can make a study possible that would otherwise be unaffordable (Kalton, 2009).
Informed answers
People who know a topic can answer detailed questions about it. Respondents who lack relevant experience may still answer, but their answers are often guesses or reflect little thought. Theories of survey response suggest that respondents who find a task meaningful and within their ability are more likely to answer carefully, while those who find it difficult or irrelevant are more prone to satisficing (Krosnick, 1991).
Higher participation
People are more likely to take part in surveys on subjects that interest them. Leverage-saliency theory explains this: different features of a survey request matter to different people, and a topic that is personally relevant makes the request more attractive (Groves et al., 2000). Research on web surveys confirms that interest in the topic is an important reason why people take part (Keusch, 2015).
What targeting does to a sample
Over-representation of the interested
The same mechanism that raises participation also biases it. Experiments in which survey invitations emphasised different topics found that people interested in the stated topic were much more likely to respond (Groves et al., 2004). When interest is related to the answers, this produces nonresponse bias. In one series of experiments, describing a survey as being about a particular subject attracted respondents whose answers on that subject differed from those of people who received a neutral description (Groves et al., 2006).
Figure 1 illustrates the effect. A survey about home fitness equipment, recruited through fitness websites and advertising, contains far more users, enthusiasts and recent buyers than the general population (illustrative figures).
This is exactly what the researcher wanted if the goal is to study fitness equipment users. It is a serious problem if the same sample is used to estimate how many adults use such equipment, because the targeted sample would overstate it many times over.
Within-group differences
Even when the target population is defined correctly, targeting can favour particular members of it. A sample of electric vehicle owners recruited through an enthusiasts' forum will over-represent the most engaged owners, who may be more satisfied, more knowledgeable or more critical than typical owners. Self-selection operates within the target group as well as between it and the rest of the population (Bethlehem, 2010).
Unknown selection
In probability screening, the researcher knows how the eligible respondents were found and can calculate their selection probabilities. In most online targeting, the selection is governed by advertising algorithms, website audiences and self-selection, and the probabilities are unknown (Baker et al., 2013).
Defining the target population
The most important step is to define precisely whom the survey is meant to describe. "People interested in home fitness" is vague. "Adults who have bought home fitness equipment in the past two years" can be checked with screening questions and compared with other data. Clear definitions make it possible to judge whether the recruited sample matches the target population and to report results honestly.
Two kinds of estimate should be kept apart:
- Estimates within the target group, such as satisfaction among product users or reasons for choosing a brand. Targeted samples are designed for these, though within-group selection still needs attention.
- Estimates of the size of the target group, such as the share of adults who use a product. Targeted samples cannot provide these; a general population sample with screening is needed (Kalton & Anderson, 1986).
Reducing bias in targeted samples
Use neutral invitations where possible
Invitations that describe the survey in general terms, rather than naming the specific topic, reduce selection by interest. Groves et al. (2004) found that topic interest affected participation most when the topic was made salient in the request. Some studies therefore describe the survey broadly, for example as a study of "household purchases", and identify eligible people through screening questions.
Diversify sources
Recruiting from several sources, rather than one community or website, reduces the influence of any single audience. Recording the source of each respondent allows comparisons and adjustment (Callegaro et al., 2014).
Measure and adjust for engagement
If interest or engagement is likely to drive both participation and answers, it should be measured, for example by asking how often respondents use the product or how much they follow the topic. Results can then be compared across levels of engagement and weighted if external information on the distribution of engagement in the target group is available (Mercer et al., 2017; Elliott & Valliant, 2017).
Calibrate with a broader sample
A general population survey with screening can provide benchmarks for the target group, such as its age profile and usage levels. The targeted sample can then be weighted to match. This combination uses the general survey for representativeness and the targeted sample for depth (Cornesse et al., 2020).
Writing screening questions
Screening questions decide who counts as part of the target group, so their design matters. They should be specific, refer to a clear time period and avoid signalling the "right" answer. A question such as "Have you bought any of the following items in the past 12 months?", followed by a list that includes the target product among several others, is better than "Have you bought home fitness equipment recently?", which reveals what the survey is looking for. When respondents are rewarded for completing surveys, some will claim eligibility they do not have in order to continue, and lists that include an item that does not exist, or questions about details that only genuine users would know, help to detect this (Callegaro et al., 2014).
The share of people who pass the screener should be recorded. In a probability survey, this share estimates the size of the target group. In a targeted nonprobability survey, it does not, because the people screened were not a random selection, but changes in the share over time or across sources can still reveal shifts in who is being recruited.
Targeting and the length of the survey
Because targeted respondents are interested in the topic, researchers are sometimes tempted to ask them many more questions. Interest does make people more willing to persist, but it does not remove the effects of length on attention and data quality. Questionnaires for targeted samples should still be kept as short as the research questions allow, and the most important questions placed early (Dillman et al., 2014).
Ethical considerations
Targeting based on personal characteristics, especially health conditions or other sensitive attributes, raises privacy questions. Researchers should be transparent about why people were invited, avoid inferences that respondents might find intrusive and follow data protection rules for any information used in targeting (Whitaker et al., 2017).
An example
A health charity wants to understand the experiences of people living with a chronic skin condition. A general population survey would include very few such people. The charity recruits through its website, patient forums and targeted social media advertising, and receives 2,500 responses. It recognises that people who visit its website and forums are likely to be more engaged with the condition and may have more severe symptoms. It asks about severity, time since diagnosis and use of support services, and compares the sample's profile with figures from a national health survey that identified people with the condition. It then weights the results to match on age, sex and severity. In its report it presents findings as describing people with the condition, discusses the likely over-representation of engaged patients and does not use the sample to estimate how common the condition is (illustrative example).
Checklist
- Define the target population in terms that can be checked by screening questions.
- Separate estimates within the target group from estimates of its size.
- Use neutral invitations where possible, and screen for eligibility.
- Recruit from several sources and record the source of each respondent.
- Measure engagement and interest, and examine how results vary with them.
- Weight to external benchmarks for the target group where they exist.
- Report the recruitment methods and their likely effects (Dillman et al., 2014).
Conclusion
Topic-targeted recruitment matches respondents to the questions asked. It makes surveys of specific groups efficient, produces better-informed answers and raises participation. The same features mean that targeted samples over-represent the interested and engaged, both relative to the general population and within the target group. These effects can be managed by defining the population carefully, keeping estimates of size separate from estimates within the group, using neutral invitations, diversifying sources and adjusting with external benchmarks. Used in this way, targeting turns limited budgets into relevant knowledge without overstating what that knowledge can show (Groves & Couper, 1998; Groves & Peytcheva, 2008).
References
Antoun, C., Zhang, C., Conrad, F. G., & Schober, M. F. (2016). Comparisons of online recruitment strategies for convenience samples: Craigslist, Google AdWords, Facebook, and Amazon Mechanical Turk. Field Methods, 28(3), 231–246. https://doi.org/10.1177/1525822X15603149
Baker, R., Brick, J. M., Bates, N. A., Battaglia, M., Couper, M. P., Dever, J. A., Gile, K. J., & Tourangeau, R. (2013). Summary report of the AAPOR task force on non-probability sampling. Journal of Survey Statistics and Methodology, 1(2), 90–143. https://doi.org/10.1093/jssam/smt008
Bethlehem, J. (2010). Selection bias in web surveys. International Statistical Review, 78(2), 161–188. https://doi.org/10.1111/j.1751-5823.2010.00112.x
Callegaro, M., Baker, R., Bethlehem, J., Göritz, A. S., Krosnick, J. A., & Lavrakas, P. J. (Eds.). (2014). Online panel research: A data quality perspective. Wiley. https://doi.org/10.1002/9781118763520
Cornesse, C., Blom, A. G., Dutwin, D., Krosnick, J. A., De Leeuw, E. D., Legleye, S., Pasek, J., Pennay, D., Phillips, B., Sakshaug, J. W., Struminskaya, B., & Wenz, A. (2020). A review of conceptual approaches and empirical evidence on probability and nonprobability sample survey research. Journal of Survey Statistics and Methodology, 8(1), 4–36. https://doi.org/10.1093/jssam/smz041
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Wiley.
Elliott, M. R., & Valliant, R. (2017). Inference for nonprobability samples. Statistical Science, 32(2), 249–264. https://doi.org/10.1214/16-STS598
Groves, R. M., & Couper, M. P. (1998). Nonresponse in household interview surveys. Wiley. https://doi.org/10.1002/9781118490082
Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72(2), 167–189. https://doi.org/10.1093/poq/nfn011
Groves, R. M., Couper, M. P., Presser, S., Singer, E., Tourangeau, R., Acosta, G. P., & Nelson, L. (2006). Experiments in producing nonresponse bias. Public Opinion Quarterly, 70(5), 720–736. https://doi.org/10.1093/poq/nfl036
Groves, R. M., Presser, S., & Dipko, S. (2004). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68(1), 2–31. https://doi.org/10.1093/poq/nfh002
Groves, R. M., Singer, E., & Corning, A. (2000). Leverage-saliency theory of survey participation: Description and an illustration. Public Opinion Quarterly, 64(3), 299–308. https://doi.org/10.1086/317990
Kalton, G., & Anderson, D. W. (1986). Sampling rare populations. Journal of the Royal Statistical Society: Series A (General), 149(1), 65–82. https://doi.org/10.2307/2981886
Kalton, G. (2009). Methods for oversampling rare subpopulations in social surveys. Survey Methodology, 35(2), 125–141.
Keusch, F. (2015). Why do people participate in Web surveys? Applying survey participation theory to Internet survey data collection. Management Review Quarterly, 65(3), 183–216. https://doi.org/10.1007/s11301-014-0111-y
Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236. https://doi.org/10.1002/acp.2350050305
Mercer, A. W., Kreuter, F., Keeter, S., & Stuart, E. A. (2017). Theory and practice in nonprobability surveys: Parallels between causal inference and survey inference. Public Opinion Quarterly, 81(S1), 250–271. https://doi.org/10.1093/poq/nfw060
Ramo, D. E., & Prochaska, J. J. (2012). Broad reach and targeted recruitment using Facebook for an online survey of young adult substance use. Journal of Medical Internet Research, 14(1), e28. https://doi.org/10.2196/jmir.1878
Whitaker, C., Stevelink, S., & Fear, N. (2017). The use of Facebook in recruiting participants for health research purposes: A systematic review. Journal of Medical Internet Research, 19(8), e290. https://doi.org/10.2196/jmir.7071