Surveys · Incentives and engagement
Survey Incentives: Do Rewards Improve Response Quality or Only Quantity?
Incentives are one of the most reliable ways to raise survey response rates. This article reviews the evidence on whether they also change who responds and how carefully people answer, and what this means for survey design.
Few findings in survey research are as consistent as the effect of incentives on response. Offering people money, a gift or a chance to win a prize increases the share who take part, in mail, telephone, face-to-face and web surveys alike (Church, 1993; Singer & Ye, 2013). Incentives are now standard in many surveys, from government studies to commercial online panels.
A higher response rate, however, is not the same as better data. Researchers and clients often ask two further questions. Do incentives change who responds, perhaps reducing nonresponse bias or perhaps introducing new bias? And do incentives change how people respond: do paid respondents answer more carefully, or do they rush through to collect their reward? This article reviews the evidence on both questions.
Why incentives work
Several theories explain why incentives increase participation. Social exchange theory holds that people respond when they expect the rewards of taking part to outweigh the costs and when they trust the organisation asking (Dillman et al., 2014). A small gift sent with the request creates a sense of obligation under the norm of reciprocity: people feel they should return a favour (Gouldner, 1960). This explains why small unconditional incentives sent in advance work better than larger ones promised on completion, a topic covered in a separate article.
Leverage-saliency theory proposes that each person weighs different features of a survey request, such as the topic, the sponsor, the burden and the incentive, and that the importance of each feature differs between people (Groves et al., 2000). An incentive gives an additional reason to take part to people who are not attracted by the topic or the sponsor. This has a direct implication for quality: if incentives bring in people who would otherwise not respond because they are not interested in the topic, they may reduce bias related to topic interest.
Effects on response rates
Meta-analyses have shown large and consistent effects. Church (1993) found that prepaid monetary incentives in mail surveys increased response rates by around 19 percentage points on average, while promised incentives and non-monetary gifts had smaller effects. Edwards et al. (2002) reviewed randomised trials of postal questionnaires and found that monetary incentives roughly doubled the odds of response. Singer et al. (1999) found that incentives also increased response in interviewer-administered surveys, though by less than in mail surveys.
The effect increases with the size of the incentive, but with diminishing returns. Mercer et al. (2015) analysed experiments in household surveys and found that each additional dollar produced a smaller gain in response. Figure 1 illustrates the pattern.
In web surveys, the evidence is more mixed. Göritz (2006) found in a meta-analysis that material incentives increased both the share of people who started a web survey and the share who finished it, but the effects were smaller than in mail surveys. Electronic incentives, such as vouchers sent by email, are less effective than cash in hand, partly because they feel less tangible and partly because they require effort to redeem.
Effects on who responds
If incentives simply bring in more of the same kind of people who would have responded anyway, they raise the response rate without reducing bias. If they bring in people who differ from regular respondents, they can reduce bias, or, in some cases, introduce it.
The evidence suggests that incentives often have larger effects among groups that are otherwise less likely to respond. Singer and Ye (2013) reviewed studies showing that incentives had larger effects among people with lower incomes, lower education and less interest in the topic or in civic participation. Groves et al. (2006) showed in experiments that people with an interest in a survey's topic were more likely to respond, and that an incentive reduced this difference. Ryu et al. (2006) found that cash and in-kind incentives attracted somewhat different respondents, indicating that the type of incentive can affect sample composition.
These findings suggest that incentives can reduce nonresponse bias, particularly bias related to interest in the topic. However, the reduction is not guaranteed. Incentives can also attract people who are motivated mainly by money, and in online panels this can encourage professional respondents and fraud. Whether an incentive reduces bias for a particular survey depends on whether the people it brings in differ from other respondents on the survey's key variables.
Effects on response quality
A common concern is that paid respondents will make less effort, answering quickly or carelessly to earn their reward. The opposite argument is also made: people who receive an incentive may feel obliged to do a good job, and those who would otherwise refuse but are persuaded by the incentive may be less motivated, lowering quality.
Most experimental evidence finds little effect of incentives on data quality, and where effects exist they tend to be small and favourable. Singer and Ye (2013) reviewed studies of item nonresponse, open-ended answers and other quality indicators and concluded that incentives seldom reduced quality and sometimes improved it. Davern et al. (2003) found no reduction in data quality from prepaid incentives in a large face-to-face survey. Medway and Tourangeau (2015) tested prepaid cash incentives in a telephone survey and found that they did not harm quality and slightly reduced some indicators of satisficing. Grauenhorst et al. (2016) found that incentives in a national face-to-face survey in Germany did not affect response quality. Figure 2 illustrates the typical pattern.
These findings fit satisficing theory, which predicts that respondents' effort depends on their motivation, the difficulty of the task and their ability (Krosnick, 1991). An incentive that increases motivation, or a sense of obligation, should if anything increase effort.
Types of incentive
Incentives differ in form as well as size, and the form affects both response and composition.
Cash is the most effective form in most studies, because it has a clear value and can be used for anything (Church, 1993). Cash-like incentives, such as cheques or bank transfers, work less well, partly because some are never cashed. Gift cards and vouchers are convenient for web surveys but are usually less effective than the same amount in cash.
Lotteries and prize draws offer a chance of winning a larger prize. They are cheap per respondent and popular in web surveys, but most studies find that they have small effects on response compared with a guaranteed incentive of similar expected cost (Göritz, 2006; Singer & Ye, 2013).
Non-monetary gifts, such as pens, calendars or small household items, have modest effects. Donations to charity made on the respondent's behalf generally do little to increase response, although some respondents value them.
Incentives in interviewer surveys and refusal conversion
In telephone and face-to-face surveys, incentives can be offered at the first approach or held back for people who initially refuse. Singer et al. (2000) found in telephone experiments that prepaid incentives sent with an advance letter increased response, and that incentives offered to refusers increased conversion rates. Targeting incentives at refusers is cheaper than offering them to everyone, but it raises questions of fairness, since cooperative respondents receive less than reluctant ones, and it may leave interviewers less motivated to persuade without the incentive.
Incentives in panel and longitudinal surveys
In panel studies, incentives are paid at each wave and become part of members' expectations. Laurie and Lynn (2009) reviewed the use of incentives in longitudinal surveys and found that they help maintain response over waves, that unconditional incentives sent before each wave are effective and that withdrawing or reducing an established incentive can harm retention. Changes to incentives in a panel should therefore be tested carefully and introduced gradually.
Incentives in online panels
Online panels are a special case. Panel members are paid, usually in points, for each survey they complete, and the incentive structure is part of the business model. Here the risk to quality is real: members who take many surveys for the reward may speed through questionnaires, and the reward attracts fraudulent accounts. Panel companies manage this through quality checks, limits on the number of surveys per member and removal of members who repeatedly fail checks (Callegaro et al., 2014).
Within a panel, increasing the reward for a particular survey usually increases response to that survey but may change its composition, attracting members who are most responsive to rewards. Because panel incentives are paid on completion, they do not create the sense of reciprocity that prepaid incentives create.
Ethical considerations
Incentives raise ethical questions. Payments that are too large could pressure people to take part in research they would otherwise decline, or to disclose information they would rather keep private. Singer and Couper (2008) tested this in experiments and found that incentives did not lead people to accept higher risks in research, which suggests that typical survey incentives do not exert undue influence. Incentives are usually regarded as a token of appreciation for respondents' time rather than as payment for a service, and they should not be so large that people feel unable to refuse.
Differential incentives, offering more to some groups than others, can improve representation but raise questions of fairness. Most researchers regard them as acceptable when they are used to improve the accuracy of results and are explained in the survey documentation.
Practical guidance
The evidence supports several practical conclusions:
- Use incentives where response is a concern. They are among the most cost-effective ways to raise response.
- Prefer prepaid, unconditional incentives in mail and address-based surveys; they produce larger effects than promised ones.
- Cash works best. Cash and cash-like incentives are more effective than gifts, lotteries or vouchers.
- Small amounts are efficient. Most of the gain comes from the first few dollars.
- Monitor quality. Check item nonresponse, speeding and straightlining by incentive condition, especially in online samples.
- Test when possible. Randomised experiments within a survey show the effect of incentives on response, composition and quality for that population.
An example
A regional agency runs a mail survey inviting households to respond online, with a paper questionnaire as a follow-up. It randomly assigns half the sample to receive a $5 bill with the first letter. Response is 41 per cent with the incentive and 28 per cent without. Respondents in the incentive group include more renters and younger adults, bringing the sample closer to census figures. Item nonresponse, straightlining and the length of open answers are similar in the two groups. Because the incentive group needed fewer reminder mailings, the cost per completed questionnaire is only slightly higher (illustrative example).
Conclusion
Incentives reliably increase survey response, and the weight of evidence shows that they do so without harming, and sometimes while improving, the quality of answers. They often bring in people who would otherwise be under-represented, which can reduce nonresponse bias, although this is not guaranteed. The main risks arise in online panels, where rewards paid per completion can attract inattentive or fraudulent respondents. Designed with these findings in mind, incentives improve both the quantity and the quality of the knowledge that surveys produce.
References
Callegaro, M., Baker, R., Bethlehem, J., Göritz, A. S., Krosnick, J. A., & Lavrakas, P. J. (Eds.). (2014). Online panel research: A data quality perspective. Wiley. https://doi.org/10.1002/9781118763520
Church, A. H. (1993). Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opinion Quarterly, 57(1), 62–79. https://doi.org/10.1086/269355
Davern, M., Rockwood, T. H., Sherrod, R., & Campbell, S. (2003). Prepaid monetary incentives and data quality in face-to-face interviews: Data from the 1996 Survey of Income and Program Participation Incentive Experiment. Public Opinion Quarterly, 67(1), 139–147. https://doi.org/10.1086/346012
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Wiley.
Edwards, P., Roberts, I., Clarke, M., DiGuiseppi, C., Pratap, S., Wentz, R., & Kwan, I. (2002). Increasing response rates to postal questionnaires: Systematic review. BMJ, 324(7347), 1183. https://doi.org/10.1136/bmj.324.7347.1183
Gouldner, A. W. (1960). The norm of reciprocity: A preliminary statement. American Sociological Review, 25(2), 161–178. https://doi.org/10.2307/2092623
Grauenhorst, T., Blohm, M., & Koch, A. (2016). Respondent incentives in a national face-to-face survey: Do they affect response quality? Field Methods, 28(3), 266–283. https://doi.org/10.1177/1525822X15612710
Göritz, A. S. (2006). Incentives in web studies: Methodological issues and a review. International Journal of Internet Science, 1(1), 58–70.
Groves, R. M., Couper, M. P., Presser, S., Singer, E., Tourangeau, R., Acosta, G. P., & Nelson, L. (2006). Experiments in producing nonresponse bias. Public Opinion Quarterly, 70(5), 720–736. https://doi.org/10.1093/poq/nfl036
Groves, R. M., Singer, E., & Corning, A. (2000). Leverage-saliency theory of survey participation: Description and an illustration. Public Opinion Quarterly, 64(3), 299–308. https://doi.org/10.1086/317990
Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236. https://doi.org/10.1002/acp.2350050305
Laurie, H., & Lynn, P. (2009). The use of respondent incentives on longitudinal surveys. In P. Lynn (Ed.), Methodology of longitudinal surveys (pp. 205–233). Wiley. https://doi.org/10.1002/9780470743874.ch12
Medway, R. L., & Tourangeau, R. (2015). Response quality in telephone surveys: Do prepaid cash incentives make a difference? Public Opinion Quarterly, 79(2), 524–543. https://doi.org/10.1093/poq/nfv011
Mercer, A., Caporaso, A., Cantor, D., & Townsend, R. (2015). How much gets you how much? Monetary incentives and response rates in household surveys. Public Opinion Quarterly, 79(1), 105–129. https://doi.org/10.1093/poq/nfu059
Ryu, E., Couper, M. P., & Marans, R. W. (2006). Survey incentives: Cash vs. in-kind; face-to-face vs. mail; response rate vs. nonresponse error. International Journal of Public Opinion Research, 18(1), 89–106. https://doi.org/10.1093/ijpor/edh089
Singer, E., & Couper, M. P. (2008). Do incentives exert undue influence on survey participation? Experimental evidence. Journal of Empirical Research on Human Research Ethics, 3(3), 49–56. https://doi.org/10.1525/jer.2008.3.3.49
Singer, E., & Ye, C. (2013). The use and effects of incentives in surveys. The Annals of the American Academy of Political and Social Science, 645(1), 112–141. https://doi.org/10.1177/0002716212458082
Singer, E., Van Hoewyk, J., & Maher, M. P. (2000). Experiments with incentives in telephone surveys. Public Opinion Quarterly, 64(2), 171–188. https://doi.org/10.1086/317761
Singer, E., Van Hoewyk, J., Gebler, N., Raghunathan, T., & McGonagle, K. (1999). The effect of incentives on response rates in interviewer-mediated surveys. Journal of Official Statistics, 15(2), 217–230.