Skip to content

Surveys · Incentives and engagement

Gamification of Surveys: Engagement Gains and Measurement Risks

Gamified surveys add game elements such as points, progress levels, stories and interactive tasks to keep respondents engaged. This article reviews what the evidence shows about their effects on engagement and data quality, and the risks they pose for measurement.

ModQu research article · 1,532 words · 16 references (APA 7th edition)

Download PDF

Online surveys compete for attention with games, videos and social media. Long grids of radio buttons can feel tedious, and bored respondents may speed, straightline or abandon the questionnaire. One response has been gamification: the use of game design elements in non-game contexts (Deterding et al., 2011). In survey research, gamification ranges from small changes, such as a progress bar presented as a journey, to questionnaires rebuilt as interactive stories or challenges. Market research agencies in particular have promoted gamified surveys as a way to improve respondent engagement (Bailey et al., 2015).

The appeal is clear, but gamification also changes how questions are presented, and in survey research presentation affects answers. This article reviews the evidence on whether gamification improves engagement and data quality, and on the measurement risks it introduces.

Why engagement matters

The case for gamification rests on a well-established link between respondents' motivation and the quality of their answers. Satisficing theory holds that when motivation is low or the task is hard, respondents take shortcuts: they choose the first acceptable answer, agree with statements, give the same answer to every item in a grid or say "don't know" (Krosnick, 1991). Galesic (2006) found that respondents who reported less interest and more burden while answering a web survey were more likely to break off, and that interest declined as the questionnaire went on. If game elements raise interest and reduce felt burden, they should, in principle, reduce satisficing and breakoff. The question is whether they do so in practice, and at what cost to measurement.

What gamification looks like

Figure 1 shows six game elements commonly used in surveys.

Six cards describing points and badges, progress and levels, narrative, challenge, interactive formats and feedback.
Figure 1. Six game elements used in surveys. Diagram by ModQu.

Some elements change only the experience around the questions, such as badges awarded for completing a section. Others change the questions themselves. A question asking how many brands of soft drink the respondent can name may become a timed challenge: "You have one minute to name as many brands as you can." A question about holiday preferences may be framed as a scenario: "Imagine you have won a trip and must pack a suitcase." Answer formats may move from radio buttons to sliders, drag-and-drop tasks or card sorts. Keusch and Zhang (2017) distinguished these "hard" forms of gamification, which add game mechanics such as points and levels, from "soft" forms, which make the questionnaire more playful through wording, design and interactivity.

Effects on engagement

Reviews of gamification in many fields have found generally positive but mixed effects on motivation and engagement, depending on the context and the people involved (Hamari et al., 2014). In surveys, respondents often report enjoying gamified questionnaires more than standard ones, and some studies find longer and richer answers to open questions. Cechanowicz et al. (2013) compared levels of gamification in a market research survey and found that more gamified versions increased participants' engagement and willingness to take part again. Harms et al. (2015) designed and tested a gamified survey and found higher enjoyment than with a conventional version, although completion took longer.

Effects on completion and response rates are smaller and less consistent. Mavletova (2015) tested a gamified longitudinal web survey among children and adolescents and found that gamification increased reported enjoyment and reduced the perceived length of the survey, but had little effect on completion or on later participation. Downes-Le Guin et al. (2012) compared a standard questionnaire with versions using rich visual design and game elements and found that the gamified versions did not improve completion or data quality; some respondents took longer and more broke off. Figure 2 illustrates the typical pattern of results.

Grouped bar chart comparing a standard and a gamified survey on enjoyment, completion and length of open answers, with the largest gain in enjoyment.
Figure 2. Typical effects of gamification on enjoyment, completion and open answers (illustrative values). Chart by ModQu.

Effects on data quality

Supporters argue that engaged respondents give more thoughtful answers. Some evidence supports this for open-ended questions, where gamified framing has produced longer and more detailed answers. For closed questions, the evidence is weaker. Keusch and Zhang (2017), in a review of studies of gamified surveys, concluded that gamification often improved respondents' enjoyment but that effects on data quality indicators, such as straightlining and item nonresponse, were small and inconsistent.

There is also a risk that game elements reduce quality. Time limits encourage quick rather than careful answers. Points awarded per question may reward speed. Elaborate graphics can slow loading on mobile devices and increase breakoffs, especially for respondents with slower connections (Downes-Le Guin et al., 2012).

Gamification on smartphones

A growing share of online surveys is completed on smartphones, and this changes what gamification can do. Interactive elements designed for a computer screen, such as drag-and-drop tasks across a wide grid, may be awkward or impossible on a small touch screen. Mavletova (2013) found that respondents using mobile devices took longer to complete web surveys and broke off more often than those using computers, even with simple designs. Heavy graphics and animations add to loading times and data use. On the other hand, some touch-based formats, such as swiping cards left or right to indicate preference, are natural on smartphones and can make short tasks quicker. Any gamified design should be tested on the devices respondents actually use.

Measurement risks

The more important concern is that gamification changes the measurement itself. Survey methodologists have shown that visual and verbal features of questions affect answers (Tourangeau et al., 2013). Gamification changes many such features at once.

Framing. A scenario or story changes the context in which a question is answered. Asking respondents to imagine themselves as a company director before rating a product may produce different answers from asking them as consumers. The difference may be desirable if the scenario is the intended context, but it makes results hard to compare with those from standard questions.

Answer formats. Sliders, drag-and-drop tasks and visual scales are not equivalent to radio buttons. Funke et al. (2011) found that sliders produced more missing data and longer response times than radio buttons, particularly among respondents with less education. Visual analogue scales and other graphic formats can shift the distribution of answers.

Feedback. Showing respondents their results or comparing them with others' answers can change answers to later questions, as respondents adjust towards what they have seen.

Selection. Respondents differ in how much they enjoy games. If gamified surveys appeal more to some groups, such as younger people, they may change who completes the survey, not only how. Mavletova (2015) noted that the appeal of game elements varied by age.

Comparability. Data from a gamified survey may not be comparable with data from earlier standard surveys, which matters for tracking studies and benchmarks.

Progress indicators: a cautionary example

Progress indicators are one of the simplest game-like features and have been studied extensively. They are intended to encourage completion by showing respondents how far they have come. A meta-analysis by Villar et al. (2013) found that standard progress indicators did not reduce breakoffs on average, that indicators showing fast early progress reduced breakoffs and that those showing slow early progress increased them. Conrad et al. (2010) reached similar conclusions. The lesson is that features intended to motivate respondents can backfire, and that their effects need to be tested rather than assumed.

Using gamification responsibly

The evidence suggests a cautious approach:

An example

A food company tests a gamified version of its quarterly brand survey. Half of a panel sample receives the standard questionnaire; the other half receives a version in which the brand-awareness question becomes a one-minute challenge and the attribute ratings become a card sort. Respondents rate the gamified version as more enjoyable, and completion is similar. However, the challenge format produces higher unprompted awareness for well-known brands and lower awareness for smaller ones, and the card sort compresses differences between brands. The company keeps the enjoyable introduction and progress design but returns to the standard formats for the tracked measures (illustrative example).

Conclusion

Gamification can make surveys more enjoyable and can encourage fuller answers to open questions. Its effects on completion and on the quality of closed-question data are smaller and less consistent than its advocates have claimed, and some features, such as time limits and elaborate graphics, can harm quality. Because game elements change the context and format of questions, they also change measurement, which limits comparability with standard surveys. Used selectively, tested experimentally and kept away from key measures, gamification can be a useful addition to survey design, but it is not a substitute for short, clear and well-designed questionnaires.

References

Bailey, P., Pritchard, G., & Kernohan, H. (2015). Gamification in market research. International Journal of Market Research, 57(1), 17–28. https://doi.org/10.2501/IJMR-2015-003

Cechanowicz, J., Gutwin, C., Brownell, B., & Goodfellow, L. (2013). Effects of gamification on participation and data quality in a real-world market research domain. In Proceedings of the First International Conference on Gameful Design, Research, and Applications (pp. 58–65). ACM. https://doi.org/10.1145/2583008.2583016

Conrad, F. G., Couper, M. P., Tourangeau, R., & Peytchev, A. (2010). The impact of progress indicators on task completion. Interacting with Computers, 22(5), 417–427. https://doi.org/10.1016/j.intcom.2010.03.001

Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design elements to gamefulness: Defining "gamification". In Proceedings of the 15th International Academic MindTrek Conference (pp. 9–15). ACM. https://doi.org/10.1145/2181037.2181040

Downes-Le Guin, T., Baker, R., Mechling, J., & Ruyle, E. (2012). Myths and realities of respondent engagement in online surveys. International Journal of Market Research, 54(5), 613–633. https://doi.org/10.2501/IJMR-54-5-613-633

Funke, F., Reips, U.-D., & Thomas, R. K. (2011). Sliders for the smart: Type of rating scale on the web interacts with educational level. Social Science Computer Review, 29(2), 221–231. https://doi.org/10.1177/0894439310376896

Galesic, M., & Bosnjak, M. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73(2), 349–360. https://doi.org/10.1093/poq/nfp031

Galesic, M. (2006). Dropouts on the web: Effects of interest and burden experienced during an online survey. Journal of Official Statistics, 22(2), 313–328.

Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamification work? A literature review of empirical studies on gamification. In Proceedings of the 47th Hawaii International Conference on System Sciences (pp. 3025–3034). IEEE. https://doi.org/10.1109/HICSS.2014.377

Harms, J., Biegler, S., Wimmer, C., Kappel, K., & Grechenig, T. (2015). Gamification of online surveys: Design process, case study, and evaluation. In J. Abascal, S. Barbosa, M. Fetter, T. Gross, P. Palanque, & M. Winckler (Eds.), Human-computer interaction – INTERACT 2015 (pp. 219–236). Springer. https://doi.org/10.1007/978-3-319-22701-6_16

Keusch, F., & Zhang, C. (2017). A review of issues in gamified surveys. Social Science Computer Review, 35(2), 147–166. https://doi.org/10.1177/0894439315608451

Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236. https://doi.org/10.1002/acp.2350050305

Mavletova, A. (2013). Data quality in PC and mobile web surveys. Social Science Computer Review, 31(6), 725–743. https://doi.org/10.1177/0894439313485201

Mavletova, A. (2015). A gamification effect in longitudinal web surveys among children and adolescents. International Journal of Market Research, 57(3), 413–438. https://doi.org/10.2501/IJMR-2015-035

Tourangeau, R., Conrad, F. G., & Couper, M. P. (2013). The science of web surveys. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780199747047.001.0001

Villar, A., Callegaro, M., & Yang, Y. (2013). Where am I? A meta-analysis of experiments on the effects of progress indicators for web surveys. Social Science Computer Review, 31(6), 744–762. https://doi.org/10.1177/0894439313497468