Surveys
Surveys glossary
Terms used in designing, sampling, fielding and checking surveys, from questionnaire wording to weighting and data quality.
A
- Acquiescence bias also: yea-saying
The tendency of some respondents to agree with statements regardless of their content, sometimes called "yea-saying". It inflates agreement on agree–disagree items and can create spurious correlations between items that share the same direction of wording. Common remedies are item-specific question formats (asking directly about the attribute rather than asking for agreement) and balanced scales that mix positively and negatively worded items. (Krosnick, 1999; Tourangeau et al., 2000)
- Anchoring vignettes
Short descriptions of hypothetical people or situations that every respondent rates on the same scale used for their self-assessment. Because the vignettes are fixed, differences in how groups rate them reveal differences in how those groups use the scale, which can then be corrected. The method addresses differential item functioning, the problem that two people with the same true level may choose different answer categories. (King et al., 2004)
- Attention check also: instructed response item
An item placed in a questionnaire to detect respondents who are not reading the questions, for example an instruction to select a specific answer. Failure rates are used to flag or remove inattentive cases. Evidence suggests such checks identify careless responding but can also irritate attentive respondents, so they are best combined with other indicators such as completion time and response patterns. (Oppenheimer et al., 2009; Meade & Craig, 2012)
C
- Careless responding also: insufficient effort responding
Answering survey items without adequate attention to their content, producing responses that do not reflect the respondent's true state. Indicators include very fast completion, long strings of identical answers, inconsistent answers to items that should agree, and failed attention checks. Careless responding adds noise and can bias estimates and correlations if it is not detected. (Meade & Craig, 2012)
- Cluster sampling
A probability design in which the population is divided into groups (clusters), such as schools or city blocks, a sample of clusters is selected, and units are surveyed within the selected clusters. It reduces travel and listing costs but usually lowers precision because units in the same cluster tend to be alike, which raises the design effect. (Kish, 1965; Lohr, 2021)
- Cognitive interviewing
A pretesting method in which respondents answer draft survey questions while thinking aloud or responding to probes about how they understood each question and arrived at their answer. It reveals problems of comprehension, recall, judgement and response mapping before a survey is fielded. It is qualitative and usually involves small purposive samples. (Willis, 2005; Tourangeau et al., 2000)
- Coverage error
Error that arises when the sampling frame does not match the target population, because some members are missing from the frame (undercoverage) or the frame includes units outside the population (overcoverage). Bias results when those missing differ from those covered on the variables being measured. It is one of the error sources in the total survey error framework. (Groves et al., 2009; Lohr, 2021)
D
- Design effect also: deff
The ratio of the variance of an estimate under a complex sample design to the variance it would have under a simple random sample of the same size. Clustering and unequal weights usually raise it above 1; effective stratification can lower it. Dividing the actual sample size by the design effect gives the effective sample size, the number of simple random cases with the same precision. (Kish, 1965; Kish, 1992)
E
- Effective sample size
The size of a simple random sample that would give the same precision as the actual sample, once clustering and weighting are taken into account. For weighting alone, Kish's approximation is the square of the sum of weights divided by the sum of squared weights. Reporting it shows how much precision a design or a weighting scheme has cost. (Kish, 1965; Kish, 1992)
I
- Imputation
Filling in missing values with plausible estimates so that complete-data methods can be used. Methods range from mean and hot-deck imputation to model-based and multiple imputation. Multiple imputation creates several completed datasets so that the extra uncertainty from missing data is reflected in standard errors. (Little & Rubin, 2019)
- Item nonresponse
Missing answers to particular questions from people who otherwise took part in a survey, for example skipping an income question. It differs from unit nonresponse, where the whole interview is missing. It is usually handled by imputation or by analysis methods that make explicit assumptions about why the data are missing. (Little & Rubin, 2019; Groves et al., 2009)
L
- Leverage-saliency theory
A theory of survey participation proposing that each feature of a survey request (topic, sponsor, incentive, burden) has a leverage, the weight a given person attaches to it, and a saliency, how prominent it is made in the request. A person's decision to take part depends on the combination, which is why the same design can attract some groups and deter others and why topic interest can produce nonresponse bias. (Groves et al., 2000; Groves et al., 2004)
- Likert scale
A multi-item measure introduced by Rensis Likert in 1932 in which respondents indicate their degree of agreement with a set of statements, and the item scores are summed or averaged into a scale score. In everyday use the term is often applied to a single agree–disagree item, which is more precisely a Likert-type item. Whether such data can be treated as interval rather than ordinal is a long-running debate. (Likert, 1932)
M
- Margin of error
The half-width of a confidence interval for a survey estimate, usually at 95% confidence. For a proportion from a simple random sample it is about 1.96 times the standard error. It reflects sampling error only: it says nothing about coverage, nonresponse or measurement error, and for nonprobability samples its meaning depends on model assumptions. (Lohr, 2021; Neyman, 1934)
- Measurement error
The difference between a recorded answer and the true value of what was meant to be measured, arising from question wording, respondents' memory and motives, interviewers, mode or data processing. Random measurement error adds noise and weakens correlations; systematic error biases estimates. (Groves & Lyberg, 2010; Tourangeau et al., 2000)
- Measurement invariance
The property that a measure assesses the same construct in the same way across groups, modes or time points. It is usually tested in steps (configural, metric, scalar) with multi-group confirmatory factor analysis. Without at least scalar invariance, differences in group means may reflect differences in how the questions work rather than true differences. (King et al., 2004)
- Missing at random also: MAR
In Rubin's classification of missing data, the condition that the probability a value is missing depends only on observed data, not on the missing value itself. Data are missing completely at random if missingness is unrelated to any data, and missing not at random if it depends on the unobserved value. Most imputation and weighting methods assume missing at random. (Little & Rubin, 2019)
- Mixed-mode survey
A survey that collects data through more than one mode, such as web and telephone, either by offering a choice, by following up nonrespondents in a second mode, or by using different modes for different parts of the interview. Mixing modes can improve coverage and response but can introduce mode effects that make answers from different modes hard to compare. (de Leeuw, 2005; Dillman et al., 2014)
- Mode effect
A difference in survey answers caused by the mode of data collection (face-to-face, telephone, mail, web, app) rather than by differences between respondents. Interviewer-administered modes tend to produce more socially desirable answers on sensitive topics; visual and aural presentation can change how response scales are used. (de Leeuw, 2005; Tourangeau & Yan, 2007)
N
- Net Promoter Score also: NPS
A customer loyalty metric based on one question asking how likely a customer is to recommend a company on a 0–10 scale. The score is the percentage of promoters (9–10) minus the percentage of detractors (0–6). It was popularised as a predictor of growth, but later research found it no better than other satisfaction and loyalty measures at predicting company revenue growth. (Reichheld, 2003; Keiningham et al., 2007)
- Nonprobability sample
A sample in which units are selected by a process that does not give every member of the population a known, non-zero chance of selection, such as opt-in online panels, quota samples or convenience samples. Estimates from such samples rely on modelling assumptions, typically that weighting or matching variables account for how participants differ from the population. (Baker et al., 2013)
- Nonresponse bias
The difference between a survey estimate and the value it would have had with full response, caused by respondents differing from nonrespondents on the variable being measured. It depends on the correlation between response propensity and the survey variable, not on the response rate alone, which is why low-response surveys can be accurate and high-response surveys can be biased. (Groves, 2006; Groves & Peytcheva, 2008)
P
- Paradata
Data about the process of collecting survey data, such as call records, timestamps, keystrokes, device type and interviewer observations. Paradata are used to monitor fieldwork, detect data quality problems, model response propensity and adjust for nonresponse. (Kreuter, 2013)
- Probability sample
A sample drawn so that every unit in the population has a known, non-zero probability of selection. These probabilities allow design-based inference: unbiased estimation with weights equal to the inverse of the selection probabilities, and measurable sampling error. Simple random, systematic, stratified and cluster sampling are all forms of probability sampling. (Neyman, 1934; Kish, 1965; Lohr, 2021)
Q
- Quota sampling
A nonprobability method in which interviewers or online systems fill target numbers (quotas) of respondents in categories such as age, sex and region so that the sample matches the population on those variables. Selection within each quota cell is not random, so the sample can still differ from the population on unmeasured characteristics. Quota polls famously failed in the 1948 US presidential election. (Baker et al., 2013; Converse, 1987)
R
- Randomized response technique
A method for asking sensitive questions in which a random device, unseen by the interviewer, decides whether the respondent answers the sensitive question or an innocuous one. Because no single answer reveals the respondent's status, people may answer more honestly, while the known probabilities still allow the prevalence of the sensitive trait to be estimated. (Warner, 1965; Tourangeau & Yan, 2007)
- Respondent-driven sampling also: RDS
A network-based method for sampling hidden or hard-to-reach populations, in which initial participants (seeds) recruit peers, who recruit further peers in successive waves. Recruitment is tracked and weighted by participants' network sizes to reduce the bias of chain referral, under assumptions that are often hard to verify in practice. (Heckathorn, 1997)
- Response rate
The proportion of eligible sample members who complete a survey. AAPOR's Standard Definitions set out several formulas that differ in how partial interviews and cases of unknown eligibility are treated. A low response rate raises the risk of nonresponse bias but does not by itself show that bias is present. (American Association for Public Opinion Research, 2023; Groves & Peytcheva, 2008)
- Responsive design
A survey design that sets out in advance how fieldwork will change in phases, based on paradata and early results, to control costs and errors; for example switching mode or raising incentives for groups with low response. Adaptive design is a related approach that tailors protocols to sample members from the start. (Groves & Heeringa, 2006)
S
- Sampling frame
The list or procedure used to identify and select members of the target population, such as an address file, a telephone number range or a customer database. The quality of a sample can be no better than its frame: units missing from the frame cannot be selected, which leads to coverage error. (Groves et al., 2009; Lohr, 2021)
- Satisficing
Giving answers that are merely acceptable rather than optimal, because carefully completing each step of the response process is demanding. Weak satisficing includes choosing the first reasonable option or agreeing with statements; strong satisficing includes straightlining, choosing "don't know" or random answering. It is more likely when the task is difficult, motivation is low and ability is limited. (Krosnick, 1991)
- Snowball sampling also: chain-referral sampling
A nonprobability method in which initial participants refer others they know who meet the study criteria, who in turn refer more. It is useful for reaching hidden or rare populations, but the sample reflects the social networks of the starting points. Respondent-driven sampling adds tracking and weighting to reduce this bias. (Heckathorn, 1997)
- Straightlining also: nondifferentiation
Giving the same answer to every item in a grid or battery of questions regardless of content. It is a common indicator of satisficing or careless responding, although identical answers can sometimes be genuine. Shorter grids, item-by-item presentation and varied item wording reduce it. (Krosnick, 1991; Meade & Craig, 2012)
- Stratified sampling
A probability design in which the population is divided into non-overlapping groups (strata) and a separate sample is drawn from each. Stratification guarantees representation of each stratum and, when strata differ on the survey variable, increases precision. Neyman (1934) showed how to allocate the sample across strata to minimise variance. (Neyman, 1934; Lohr, 2021)
- Survey response process
The model of answering a survey question as four components: comprehension of the question, retrieval of relevant information from memory, judgement to combine that information into an answer, and response, mapping the answer onto the categories offered. Errors can enter at each stage; the model underpins cognitive interviewing and much questionnaire design advice. (Tourangeau et al., 2000)
T
- Telescoping
A recall error in which respondents report events as having happened more recently (forward telescoping) or longer ago (backward telescoping) than they did, moving them into or out of the reference period of a question. Bounded interviewing, where a previous interview marks the start of the period, was developed to reduce it. (Neter & Waksberg, 1964; Tourangeau et al., 2000)
- Total survey error also: TSE
A framework that treats survey quality as the sum of all errors affecting an estimate: coverage, sampling, nonresponse and adjustment errors on the representation side, and validity, measurement and processing errors on the measurement side. It encourages designers to balance errors against costs rather than focusing on sampling error alone. (Groves & Lyberg, 2010; Groves et al., 2009)
U
- Unit nonresponse
The failure to obtain any survey data from a sampled unit, because the person or organisation could not be contacted, refused, or was unable to take part. It is distinguished from item nonresponse and is the main source of declining response rates in household surveys. (Groves & Couper, 1998; Brick, 2013)
W
- Weighting also: survey weights
Assigning each respondent a weight so that weighted estimates represent the target population. Base weights correct for unequal selection probabilities; nonresponse and calibration adjustments (post-stratification, raking, generalised regression) make weighted totals match known population figures. Weighting can reduce bias but usually increases variance. (Kalton & Flores-Cervantes, 2003; Deville & Särndal, 1992; Horvitz & Thompson, 1952)
References
American Association for Public Opinion Research. (2023). Standard definitions: Final dispositions of case codes and outcome rates for surveys (10th ed.). https://aapor.org/wp-content/uploads/2023/05/Standards-Definitions-10th-edition.pdf
Baker, R., Brick, J. M., Bates, N. A., Battaglia, M., Couper, M. P., Dever, J. A., Gile, K. J., & Tourangeau, R. (2013). Summary report of the AAPOR task force on non-probability sampling. Journal of Survey Statistics and Methodology, 1(2), 90–143. https://doi.org/10.1093/jssam/smt008
Brick, J. M. (2013). Unit nonresponse and weighting adjustments: A critical review. Journal of Official Statistics, 29(3), 329–353. https://doi.org/10.2478/jos-2013-0026
Converse, J. M. (1987). Survey research in the United States: Roots and emergence, 1890–1960. University of California Press.
de Leeuw, E. D. (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21(2), 233–255.
Deville, J.-C., & Särndal, C.-E. (1992). Calibration estimators in survey sampling. Journal of the American Statistical Association, 87(418), 376–382. https://doi.org/10.1080/01621459.1992.10475217
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Wiley. https://doi.org/10.1002/9781394260645
Groves, R. M., & Couper, M. P. (1998). Nonresponse in household interview surveys. Wiley. https://doi.org/10.1002/9781118490082
Groves, R. M., & Heeringa, S. G. (2006). Responsive design for household surveys: Tools for actively controlling survey errors and costs. Journal of the Royal Statistical Society: Series A (Statistics in Society), 169(3), 439–457. https://doi.org/10.1111/j.1467-985X.2006.00423.x
Groves, R. M., & Lyberg, L. (2010). Total survey error: Past, present, and future. Public Opinion Quarterly, 74(5), 849–879. https://doi.org/10.1093/poq/nfq065
Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72(2), 167–189. https://doi.org/10.1093/poq/nfn011
Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70(5), 646–675. https://doi.org/10.1093/poq/nfl033
Groves, R. M., Fowler, F. J., Jr., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology (2nd ed.). Wiley.
Groves, R. M., Presser, S., & Dipko, S. (2004). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68(1), 2–31. https://doi.org/10.1093/poq/nfh002
Groves, R. M., Singer, E., & Corning, A. (2000). Leverage-saliency theory of survey participation. Public Opinion Quarterly, 64(3), 299–308. https://doi.org/10.1086/317990
Heckathorn, D. D. (1997). Respondent-driven sampling: A new approach to the study of hidden populations. Social Problems, 44(2), 174–199. https://doi.org/10.2307/3096941
Horvitz, D. G., & Thompson, D. J. (1952). A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260), 663–685. https://doi.org/10.1080/01621459.1952.10483446
Kalton, G., & Flores-Cervantes, I. (2003). Weighting methods. Journal of Official Statistics, 19(2), 81–98.
Keiningham, T. L., Cooil, B., Andreassen, T. W., & Aksoy, L. (2007). A longitudinal examination of net promoter and firm revenue growth. Journal of Marketing, 71(3), 39–51. https://doi.org/10.1509/jmkg.71.3.039
King, G., Murray, C. J. L., Salomon, J. A., & Tandon, A. (2004). Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98(1), 191–207. https://doi.org/10.1017/S000305540400108X
Kish, L. (1965). Survey sampling. Wiley.
Kish, L. (1992). Weighting for unequal Pi. Journal of Official Statistics, 8(2), 183–200.
Kreuter, F. (Ed.). (2013). Improving surveys with paradata: Analytic uses of process information. Wiley. https://doi.org/10.1002/9781118596869
Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236. https://doi.org/10.1002/acp.2350050305
Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537–567. https://doi.org/10.1146/annurev.psych.50.1.537
Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 22(140), 1–55.
Little, R. J. A., & Rubin, D. B. (2019). Statistical analysis with missing data (3rd ed.). Wiley. https://doi.org/10.1002/9781119482260
Lohr, S. L. (2021). Sampling: Design and analysis (3rd ed.). CRC Press. https://doi.org/10.1201/9780429298899
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085
Neter, J., & Waksberg, J. (1964). A study of response errors in expenditures data from household interviews. Journal of the American Statistical Association, 59(305), 18–55. https://doi.org/10.1080/01621459.1964.10480699
Neyman, J. (1934). On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection. Journal of the Royal Statistical Society, 97(4), 558–625. https://doi.org/10.2307/2342192
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872. https://doi.org/10.1016/j.jesp.2009.03.009
Reichheld, F. F. (2003). The one number you need to grow. Harvard Business Review, 81(12), 46–54.
Tourangeau, R., & Yan, T. (2007). Sensitive questions in surveys. Psychological Bulletin, 133(5), 859–883. https://doi.org/10.1037/0033-2909.133.5.859
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press. https://doi.org/10.1017/CBO9780511819322
Warner, S. L. (1965). Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association, 60(309), 63–69. https://doi.org/10.1080/01621459.1965.10480775
Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. SAGE. https://doi.org/10.4135/9781412983655
The tendency to answer in ways that present oneself favourably, over-reporting approved behaviours such as voting and under-reporting disapproved ones such as drug use. It is strongest with interviewers present and on sensitive topics. Self-administration, assurances of confidentiality and indirect techniques such as randomized response reduce it. (Tourangeau & Yan, 2007)
Nonresponse, weighting and data quality
See also: Randomized response technique, Mode effect, Say–do gap (Insights)
Read more: Social Desirability Bias and Techniques for Reducing It · Asking Sensitive Questions with Care and Precision