Surveys · Sampling
The Role of Social Media Recruitment in Nonprobability Survey Knowledge
Social media advertising has become a common way to recruit survey respondents, especially for specific or hard-to-reach groups. This article explains how social media recruitment works, what the evidence shows about the samples it produces, the main sources of error and how researchers can use it responsibly.
Social media platforms have changed how researchers find survey respondents. A study that once needed a mailing list, a telephone sample or a panel provider can now place advertisements in the feeds of people who match chosen characteristics, often within hours and at modest cost. Social media recruitment has been used to survey young adults, migrants, people with particular health conditions, members of minority groups and the general public in countries without good sampling frames (Kosinski et al., 2015; Thornton et al., 2016; Zindel, 2023).
The samples produced in this way are nonprobability samples. People are not selected with known probabilities from a defined frame. They see an advertisement because a platform's delivery system shows it to them, and they take part because they choose to. This affects what the resulting data can and cannot tell us (Baker et al., 2013). This article explains how social media recruitment works, reviews the evidence on the samples it produces, describes the main sources of error and sets out practices that help researchers turn social media samples into useful knowledge.
How social media recruitment works
Most social media recruitment uses paid advertising. The researcher creates an advertisement, usually a short message and an image, that invites people to take part in a study and links to an online questionnaire. The platform's tools allow the researcher to choose a target audience by location, age, sex, language and, in some cases, interests or behaviours inferred from platform activity. The researcher sets a budget, and the platform decides when and to whom the advertisement is shown (Kosinski et al., 2015; Kühne & Zindel, 2020).
Recruitment can also be unpaid. Researchers post invitations on their own pages, ask organisations to share them or post in groups and forums where members of the target population gather. Unpaid recruitment is cheaper but gives the researcher less control over who sees the invitation and is more likely to reach people connected to the researcher's own networks (Topolovec-Vranic & Natarajan, 2016).
Figure 1 shows the main steps of a paid recruitment campaign.
Why researchers use it
Social media recruitment has several advantages.
Reach. Large shares of adults in many countries use at least one social media platform, so advertising can reach a broad population (Auxier & Anderson, 2021). In countries where telephone or address frames are poor, social media may be one of few ways to reach the public quickly (Grow et al., 2020).
Targeting. Platforms allow advertisements to be shown to people in specific places, age groups or language groups. This helps when the target population is small or dispersed. Pötzschke and Braun (2017) used Facebook advertisements to recruit Polish migrants in four European countries, a group for which no sampling frame existed. Iannelli et al. (2020) used Facebook targeting to reach niche and difficult-to-reach populations and assessed the efficiency of the approach.
Speed and cost. Campaigns can start within a day and collect hundreds or thousands of responses within a week. Several reviews found that the cost per recruited participant was often lower than for traditional methods, although costs varied widely (Thornton et al., 2016; Whitaker et al., 2017).
Rapid response. During the COVID-19 pandemic, researchers used Facebook advertising to run cross-national surveys of behaviours and attitudes within weeks, filling gaps while other surveys were being adapted (Grow et al., 2020; Perrotta et al., 2021).
What the samples look like
Evidence on the composition of social media samples is consistent in some respects. Samples recruited through advertising tend to differ from the population in age, sex, education and interests, and the direction of the difference depends on the platform, the target and the content of the advertisement (Whitaker et al., 2017; Zindel, 2023). Women are often over-represented in samples recruited through Facebook, and people with more education or greater interest in the topic are more likely to respond (Thornton et al., 2016; Schneider & Harknett, 2022).
Comparisons between recruitment channels show that each produces a different sample. Antoun et al. (2016) compared Craigslist, Google advertising, Facebook and Amazon Mechanical Turk and found that the samples differed in demographics, cost and data quality. Stern et al. (2017) found demographic and data quality differences between respondents recruited through Google and Facebook. Ramo and Prochaska (2012) found that targeted Facebook advertising could recruit young adults at low cost, but the sample over-represented some groups compared with national data. Fenner et al. (2012) reported that Facebook advertising reached young women in Australia efficiently and produced a sample broadly similar to the population on several health measures, though not on all.
Schneider and Harknett (2022) recruited tens of thousands of workers at large retail and food service firms through targeted Facebook advertising. They found that, after weighting, their estimates of several characteristics were close to those from benchmark surveys, which suggests that social media samples can produce useful estimates when recruitment is carefully designed and adjusted. Results were weaker for characteristics strongly related to the likelihood of responding.
Sources of error
Social media recruitment introduces several errors that need attention.
Coverage
Only people who use the platform can be reached. Platform users differ from non-users, and the differences vary between platforms and countries. Older adults, people with less education and people in rural areas are less likely to use some platforms (Auxier & Anderson, 2021). People who use a platform rarely are less likely to see an advertisement than heavy users. Coverage also depends on how the platform estimates characteristics such as age and location. These estimates come from what users report and how they behave online, and they can be wrong, so some people outside the target audience will see an advertisement and some inside it will not. Screening questions in the survey itself are needed to confirm eligibility.
Algorithmic delivery
The platform, not the researcher, decides who among the target audience sees an advertisement. Delivery systems are designed to show advertisements to people judged most likely to click, and they can skew delivery by sex, age and other characteristics even when targeting is broad (Ali et al., 2019; Lambrecht & Tucker, 2019). Because delivery adapts to who responds, a sample can drift during a campaign towards whatever kind of person responded first. Running separate campaigns with fixed budgets for each demographic cell is a widely used remedy (Zhang et al., 2020; Neundorf & Öztürk, 2023).
Self-selection
Of those who see an advertisement, few click and fewer complete the survey. People who take part may be more interested in the topic, more trusting of research or more attracted by an incentive. Self-selection on interest in the topic is particularly serious, because it is directly related to many survey measures (Baker et al., 2013).
Cost and composition
Some groups cost far more to reach than others, and a fixed budget will tend to fill the cheapest cells first. Figure 2 illustrates how cost per completed survey can vary.
Fraud and inattentive responding
Open online surveys attract fraudulent responses: people who complete the survey several times for an incentive, people who misrepresent themselves to qualify and automated programs, often called bots. Pozzar et al. (2020) reported that a large share of responses to a survey advertised on social media came from bots and other bad actors, detected through inconsistent answers, implausible completion times and suspicious patterns. Griffin et al. (2022) described similar problems and set out procedures for protecting data integrity. Fraud risk rises with the value of incentives and with the visibility of the survey link.
Making social media samples useful
Researchers can take several steps to improve the quality of knowledge from social media recruitment.
Plan cells and budgets. Define the demographic and regional cells that need responses and give each its own campaign and budget. This stops the delivery system from concentrating on the cheapest or most responsive groups and makes it easier to fill quotas (Zhang et al., 2020; Neundorf & Öztürk, 2023).
Write neutral advertisements. Advertisement text and images influence who clicks and to whom the platform shows the advertisement. Images that appeal to one group can skew delivery (Ali et al., 2019). Neutral wording that does not reveal the expected answers also reduces self-selection on attitudes. Testing several versions and comparing the samples they produce helps identify a balanced design (Kühne & Zindel, 2020).
Protect data quality. Use several checks together: screening questions, attention checks, consistency checks, completion time limits, restrictions on repeated participation and review of open-ended answers. No single check is enough, and checks should be applied in a way that does not exclude genuine respondents who read quickly or answer unusually (Griffin et al., 2022).
Weight carefully. Weight the sample to population totals on variables known to differ between respondents and the population. Where possible, include variables related to platform use and topic interest. More advanced methods use a reference probability sample to estimate pseudo-weights or model outcomes (Elliott & Valliant, 2017; Mercer et al., 2017).
Report transparently. Describe the platform, targeting settings, advertisement content, dates, budget, number of impressions and clicks, completion and exclusion counts, and the weighting method. These details allow readers to judge how the sample was formed (Baker et al., 2013; Zindel, 2023).
Treat ethics as part of design. Social media recruitment raises questions about privacy, consent and the use of platform data. Researchers should explain the study clearly, avoid collecting platform identifiers unless needed, and consider how targeting categories based on sensitive characteristics may be seen by those targeted (Kosinski et al., 2015; Townsend & Wallace, 2016).
Choosing a platform
Platforms differ in who uses them, how advertising is targeted and how surveys can be linked. A platform popular with younger adults will reach older adults poorly, and a professional networking site will reach employed people with higher education more readily than others. The choice of platform is therefore a choice about coverage. Using more than one platform can broaden coverage, but each platform's sample should be tracked separately so that differences can be seen and, if needed, adjusted (Antoun et al., 2016; Stern et al., 2017).
Targeting options also change over time. Platforms have removed or restricted some targeting categories, particularly those related to health, ethnicity, religion and political views, in response to concerns about privacy and discrimination. Researchers should check what options are available before planning a design that depends on them and should not assume that a targeting approach used in a published study can still be repeated (Neundorf & Öztürk, 2023; Zindel, 2023).
Incentives
Many social media studies offer an incentive, such as entry into a prize draw, a gift card or a donation to charity. Incentives can increase completion, especially among groups with less interest in the topic, and may reduce self-selection on interest. They also raise the risk of fraud, because a visible reward attracts people and programs that complete surveys only to collect it (Pozzar et al., 2020; Griffin et al., 2022). Smaller incentives, prize draws instead of guaranteed payments and incentives paid only after quality checks can balance these effects. Whatever is offered should be stated clearly in the advertisement and on the consent page.
In business and market research
Social media recruitment is also used to reach specific groups of consumers and professionals, such as owners of particular products, parents of young children or people working in a given occupation. The same issues apply. Narrow targeting can be expensive, as Figure 2 illustrates, and people who claim to belong to a group in order to qualify are a particular risk when eligibility depends on self-description. Screening questions that check knowledge a genuine member would have, without revealing the eligibility rule, help reduce false claims. Because consumer and professional populations rarely have published totals for weighting, researchers should be cautious about presenting results as estimates for the whole group and should state clearly how the sample was obtained (Baker et al., 2013).
An example
A research team wants to study how adults aged 18 to 65 in a country plan for retirement, with results by age group, sex and region. It sets up 24 cells (three age groups, two sexes and four regions) and runs a separate advertising campaign for each, with a daily budget set from expected costs. Two advertisement versions with neutral images and the text "Share your views in a 10-minute study about money and the future" are tested in the first two days, and the version that produces the more balanced sample by education is kept.
Responses pass through an eligibility screener, a consent page, two attention checks and a check for duplicate devices. About 8 per cent of completed questionnaires are excluded for quality reasons. Cells for older men in rural regions fill slowly, so their budgets are increased. The final sample of 3,200 is weighted to population totals by age, sex, region and education. The team reports the platform, settings, delivery statistics, exclusions and weighting method, and compares several key results with an official survey. The comparison shows close agreement on pension membership but higher reported savings in the social media sample, which the team reports as a likely effect of self-selection on financial interest (illustrative example).
What social media samples can and cannot tell us
Social media recruitment is well suited to some purposes. It can provide rapid information, test questionnaires, reach specific populations for which no frame exists and support studies of relationships between variables, such as whether a particular attitude is associated with a behaviour. Relationships are often more robust to selection than population levels, although not always (Kosinski et al., 2015; Cornesse et al., 2020).
It is less suited to estimating precise population levels, such as the exact share of adults who hold a view, especially when the topic is related to the likelihood of responding. In those cases, social media samples should be adjusted carefully, compared with benchmarks and, where possible, combined with probability samples (Elliott & Valliant, 2017; Cornesse et al., 2020).
Conclusion
Social media recruitment gives researchers a fast, flexible and often inexpensive way to reach survey respondents, including groups that other methods struggle to find. The samples it produces are nonprobability samples, shaped by platform use, algorithmic delivery, self-selection and cost. Planning campaigns by cell, writing neutral advertisements, checking data quality, weighting carefully and reporting transparently can make these samples a sound source of knowledge for many purposes. Recognising their limits, particularly for estimating population levels, keeps that knowledge in proportion.
References
Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., & Rieke, A. (2019). Discrimination through optimization: How Facebook's ad delivery can lead to biased outcomes. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), Article 199. https://doi.org/10.1145/3359301
Antoun, C., Zhang, C., Conrad, F. G., & Schober, M. F. (2016). Comparisons of online recruitment strategies for convenience samples: Craigslist, Google AdWords, Facebook, and Amazon Mechanical Turk. Field Methods, 28(3), 231–246. https://doi.org/10.1177/1525822X15603149
Auxier, B., & Anderson, M. (2021). Social media use in 2021. Pew Research Center.
Baker, R., Brick, J. M., Bates, N. A., Battaglia, M., Couper, M. P., Dever, J. A., Gile, K. J., & Tourangeau, R. (2013). Summary report of the AAPOR task force on non-probability sampling. Journal of Survey Statistics and Methodology, 1(2), 90–143. https://doi.org/10.1093/jssam/smt008
Cornesse, C., Blom, A. G., Dutwin, D., Krosnick, J. A., De Leeuw, E. D., Legleye, S., Pasek, J., Pennay, D., Phillips, B., Sakshaug, J. W., Struminskaya, B., & Wenz, A. (2020). A review of conceptual approaches and empirical evidence on probability and nonprobability sample survey research. Journal of Survey Statistics and Methodology, 8(1), 4–36. https://doi.org/10.1093/jssam/smz041
Elliott, M. R., & Valliant, R. (2017). Inference for nonprobability samples. Statistical Science, 32(2), 249–264. https://doi.org/10.1214/16-STS598
Fenner, Y., Garland, S. M., Moore, E. E., Jayasinghe, Y., Fletcher, A., Tabrizi, S. N., Gunasekaran, B., & Wark, J. D. (2012). Web-based recruiting for health research using a social networking site: An exploratory study. Journal of Medical Internet Research, 14(1), e20. https://doi.org/10.2196/jmir.1978
Griffin, M., Martino, R. J., LoSchiavo, C., Comer-Carruthers, C., Krause, K. D., Stults, C. B., & Halkitis, P. N. (2022). Ensuring survey research data integrity in the era of internet bots. Quality & Quantity, 56, 2841–2852. https://doi.org/10.1007/s11135-021-01252-1
Grow, A., Perrotta, D., Del Fava, E., Cimentada, J., Rampazzo, F., Gil-Clavel, S., & Zagheni, E. (2020). Addressing public health emergencies via Facebook surveys: Advantages, challenges, and practical considerations. Journal of Medical Internet Research, 22(12), e20653. https://doi.org/10.2196/20653
Iannelli, L., Giglietto, F., Rossi, L., & Zurovac, E. (2020). Facebook digital traces for survey research: Assessing the efficiency and effectiveness of a Facebook ad–based procedure for recruiting online survey respondents in niche and difficult-to-reach populations. Social Science Computer Review, 38(4), 462–476. https://doi.org/10.1177/0894439318816638
Kühne, S., & Zindel, Z. (2020). Using Facebook and Instagram to recruit web survey participants: A step-by-step guide and application. Survey Methods: Insights from the Field. https://doi.org/10.13094/SMIF-2020-00017
Kosinski, M., Matz, S. C., Gosling, S. D., Popov, V., & Stillwell, D. (2015). Facebook as a research tool for the social sciences: Opportunities, challenges, ethical considerations, and practical guidelines. American Psychologist, 70(6), 543–556. https://doi.org/10.1037/a0039210
Lambrecht, A., & Tucker, C. (2019). Algorithmic bias? An empirical study of apparent gender-based discrimination in the display of STEM career ads. Management Science, 65(7), 2966–2981. https://doi.org/10.1287/mnsc.2018.3093
Mercer, A. W., Kreuter, F., Keeter, S., & Stuart, E. A. (2017). Theory and practice in nonprobability surveys: Parallels between causal inference and survey inference. Public Opinion Quarterly, 81(S1), 250–271. https://doi.org/10.1093/poq/nfw060
Neundorf, A., & Öztürk, A. (2023). How to improve representativeness and cost-effectiveness in samples recruited through Meta: A comparison of advertisement tools. PLOS ONE, 18(2), e0281243. https://doi.org/10.1371/journal.pone.0281243
Perrotta, D., Grow, A., Rampazzo, F., Cimentada, J., Del Fava, E., Gil-Clavel, S., & Zagheni, E. (2021). Behaviours and attitudes in response to the COVID-19 pandemic: Insights from a cross-national Facebook survey. EPJ Data Science, 10, Article 17. https://doi.org/10.1140/epjds/s13688-021-00270-1
Pozzar, R., Hammer, M. J., Underhill-Blazey, M., Wright, A. A., Tulsky, J. A., Hong, F., Gundersen, D. A., & Berry, D. L. (2020). Threats of bots and other bad actors to data quality following research participant recruitment through social media: Cross-sectional questionnaire. Journal of Medical Internet Research, 22(10), e23021. https://doi.org/10.2196/23021
Pötzschke, S., & Braun, M. (2017). Migrant sampling using Facebook advertisements: A case study of Polish migrants in four European countries. Social Science Computer Review, 35(5), 633–653. https://doi.org/10.1177/0894439316666262
Ramo, D. E., & Prochaska, J. J. (2012). Broad reach and targeted recruitment using Facebook for an online survey of young adult substance use. Journal of Medical Internet Research, 14(1), e28. https://doi.org/10.2196/jmir.1878
Schneider, D., & Harknett, K. (2022). What's to like? Facebook as a tool for survey data collection. Sociological Methods & Research, 51(1), 108–140. https://doi.org/10.1177/0049124119882477
Stern, M. J., Bilgen, I., McClain, C., & Hunscher, B. (2017). Effective sampling from social media sites and search engines for web surveys: Demographic and data quality differences in surveys of Google and Facebook users. Social Science Computer Review, 35(6), 713–732. https://doi.org/10.1177/0894439316683344
Thornton, L., Batterham, P. J., Fassnacht, D. B., Kay-Lambkin, F., Calear, A. L., & Hunt, S. (2016). Recruiting for health, medical or psychosocial research using Facebook: Systematic review. Internet Interventions, 4, 72–81. https://doi.org/10.1016/j.invent.2016.02.001
Topolovec-Vranic, J., & Natarajan, K. (2016). The use of social media in recruitment for medical research studies: A scoping review. Journal of Medical Internet Research, 18(11), e286. https://doi.org/10.2196/jmir.5698
Townsend, L., & Wallace, C. (2016). Social media research: A guide to ethics. University of Aberdeen.
Whitaker, C., Stevelink, S., & Fear, N. (2017). The use of Facebook in recruiting participants for health research purposes: A systematic review. Journal of Medical Internet Research, 19(8), e290. https://doi.org/10.2196/jmir.7071
Zhang, B., Mildenberger, M., Howe, P. D., Marlon, J., Rosenthal, S. A., & Leiserowitz, A. (2020). Quota sampling using Facebook advertisements. Political Science Research and Methods, 8(3), 558–564. https://doi.org/10.1017/psrm.2018.49
Zindel, Z. (2023). Social media recruitment in online survey research: A systematic literature review. Methods, Data, Analyses, 17(2), 207–248. https://doi.org/10.12758/mda.2022.15