Surveys · Questionnaire design
Recall Bias and Telescoping in Retrospective Survey Questions
Questions about the past rely on memory, and memory makes predictable errors. This article explains recall bias and telescoping, reviews the classic evidence and describes the techniques survey designers use to reduce these errors.
Many survey questions ask about the past. "In the past twelve months, how many times did you stay overnight in hospital?" "When did you last change your mobile phone provider?" "In the past six months, has anyone broken into your home?" Each of these questions depends on the respondent's memory, and memory is neither complete nor precise. Some events are forgotten. Others are remembered but placed at the wrong time. And memories can be reshaped by what people know or believe now.
These problems are known collectively as recall bias. One specific form, telescoping, in which events are misdated so that they fall inside or outside the period the question asks about, has been studied in detail since the 1960s. This article explains how these errors arise, summarises the key evidence and describes the techniques survey designers use to reduce them.
How memory fails
Memory researchers have described several ways in which memory systematically departs from what actually happened (Schacter, 1999). Three are especially relevant to surveys.
Forgetting. Memories fade over time. The longer ago an event happened, and the less distinctive or important it was, the more likely it is to be forgotten. Routine events, such as a particular shopping trip or a visit to a family doctor, are especially hard to recall individually, because similar events merge in memory (Means & Loftus, 1991). In surveys, forgetting leads to under-reporting, which grows with the length of the reference period (Sudman & Bradburn, 1973).
Dating errors. Even when an event is remembered, its date is often uncertain. People rarely store the date of an event directly; instead they infer it from cues such as the season, other events happening at the time or how vivid the memory feels (Thompson et al., 1996). These inferences are imprecise, and the imprecision grows with time.
Reconstruction. Memories are rebuilt each time they are recalled, using current knowledge and beliefs to fill gaps. Michael Ross showed that people reconstruct their past attitudes and behaviour using implicit theories about whether they have changed. Someone who believes their views are stable will tend to recall past views as similar to present ones, even when they were different (Ross, 1989). Bradburn and colleagues showed that answers to autobiographical survey questions frequently rely on such inference rather than on direct recall (Bradburn et al., 1987).
Telescoping
Telescoping is a dating error in which events are placed at the wrong time relative to the reference period.
Forward telescoping, the more common form, occurs when an event that happened before the reference period is remembered as more recent and is wrongly included. A respondent asked about the past six months may report a car repair that actually took place eight months ago. Backward telescoping occurs when a recent event is remembered as older and is wrongly excluded.
The classic study of telescoping was carried out by John Neter and Joseph Waksberg for the United States Census Bureau, using a survey of household spending on home repairs and improvements (Neter & Waksberg, 1964). They compared reports from interviews in which the start of the reference period had been marked by an earlier interview with reports from interviews in which it had not. Without this marker, respondents reported substantially more jobs and spending for the most recent period, because jobs from before the period were being pulled forward into it. The comparison showed that telescoping could inflate estimates considerably, particularly for larger and more memorable jobs.
Why does forward telescoping happen? One explanation is that the uncertainty in dating an event increases with time. Older events are dated less precisely, so their estimated dates scatter more widely. Because there are more older events than recent ones that can be wrongly moved across the boundary, and because dates cannot be placed in the future, the net effect is that more events are moved forward into the reference period than are moved out of it (Rubin & Baddeley, 1989). Another explanation is that vivid memories seem recent: events that are easy to recall are judged to have happened more recently than they did. Respondents also tend to round dates to convenient intervals, such as "about a month ago" or "about a year ago," which adds further error (Huttenlocher et al., 1990).
Telescoping and forgetting work in opposite directions. Forgetting reduces reports and grows with longer reference periods; telescoping increases reports and is proportionally larger for shorter periods, where a few misplaced events make a bigger difference. The net error therefore depends on the length of the period and the type of event (Sudman & Bradburn, 1973).
Recall bias between groups
Recall errors become especially serious when they differ between the groups being compared. In medical research, case-control studies compare people with a disease and people without it, and ask both groups about past exposures, such as diet, medication or chemicals. People with a disease have often thought hard about what might have caused it, and they may recall past exposures more completely, or even over-report them, compared with healthy controls. This differential recall can create an apparent link between an exposure and a disease where none exists (Raphael, 1987; Coughlin, 1990).
The same problem can arise in business surveys. Customers who have recently had a bad experience with a product may recall earlier problems more readily than satisfied customers, which can exaggerate the apparent link between those problems and dissatisfaction. Customers who have just switched to a competitor may reconstruct their reasons for switching in a way that justifies the decision.
Reducing recall errors
Bounding
Bounded recall uses an earlier interview to mark the start of the reference period. In a panel survey, the interviewer can remind the respondent of events reported last time and ask only about events since then, so that earlier events are not reported again. This was the technique tested by Neter and Waksberg, and it remains standard in surveys such as national crime victimisation surveys. In one-off surveys, a stated start date ("since 1 March") provides a weaker form of bounding.
Landmark events
A memorable public or personal event can serve as a boundary. Elizabeth Loftus and Wesley Marburger asked respondents about crimes they had experienced either "in the past six months" or "since the eruption of Mt. St. Helens," a dramatic event that had occurred about six months earlier. Respondents given the landmark reported fewer incidents, consistent with reduced forward telescoping (Loftus & Marburger, 1983). Landmarks are not perfect, since the dates of public events can themselves be telescoped (Gaskell et al., 2000), but personal landmarks such as birthdays, holidays or the start of a school year are often effective.
Shorter reference periods
Shorter periods reduce forgetting, though they increase the relative impact of telescoping. The best choice depends on how often the behaviour occurs: short periods for frequent, routine events and longer periods for rare, memorable ones (Sudman & Bradburn, 1973).
Calendars and timelines
Event history calendars present the reference period as a visual timeline on which respondents record major life events, such as moves, job changes and births, and then place other events relative to them. By drawing on the way autobiographical memory is organised, calendars help respondents retrieve and date events more accurately (Belli, 1998). An experimental comparison found that calendar interviewing produced better-quality retrospective reports than standard question lists for several domains (Belli et al., 2001).
Decomposition and cues
Breaking a broad question into narrower ones, or providing examples of the kinds of events that count, helps respondents retrieve events they might otherwise miss (Means & Loftus, 1991). Asking respondents to work backwards from the present, or through the period month by month, can also help.
Records and real-time measurement
The most reliable way to avoid recall error is to avoid relying on memory. Respondents can be encouraged to consult bank statements, receipts, medical cards or app histories. Diaries capture events as they happen. Methods that ask brief questions several times a day on a smartphone collect information close to the moment of experience (Shiffman et al., 2008), and the day reconstruction method asks respondents to rebuild only the previous day, episode by episode (Kahneman et al., 2004).
Checking for recall error
Recall error can often be detected even when it cannot be eliminated. Several checks are useful:
- Compare reference periods. If the number of events reported for the most recent month is much higher than for earlier months of the same length, forward telescoping or forgetting of older events is likely.
- Look for heaping in dates. Reports that cluster at "one month ago," "six months ago" or "a year ago" show that respondents are estimating dates rather than recalling them.
- Validate against records. For a sample of respondents who give consent, comparing survey reports with administrative or company records shows the size and direction of recall error. Neter and Waksberg's study is an early example of this approach.
- Compare groups on neutral items. In studies that compare groups, such as satisfied and dissatisfied customers, asking both groups about past events that should not differ between them can reveal differential recall.
Reports of recall error should be part of the documentation of any survey that relies heavily on retrospective questions, so that users can judge how much confidence to place in the figures.
Conclusion
Retrospective survey questions rely on memory, and memory makes predictable errors. Events are forgotten, especially routine ones and those further in the past. Remembered events are misdated, most often by being pulled forward into the reference period. Past attitudes and behaviours are reconstructed in the light of present beliefs. When these errors differ between groups, they can create false conclusions. Survey designers can reduce them by bounding the reference period, using landmarks, choosing period lengths that suit the behaviour, supporting memory with calendars and cues, and replacing recall with records or real-time measurement where possible (Tourangeau et al., 2000; Biemer & Lyberg, 2003).
References
Belli, R. F. (1998). The structure of autobiographical memory and the event history calendar: Potential improvements in the quality of retrospective reports in surveys. Memory, 6(4), 383–406. https://doi.org/10.1080/741942610
Belli, R. F., Shay, W. L., & Stafford, F. P. (2001). Event history calendars and question list surveys: A direct comparison of interviewing methods. Public Opinion Quarterly, 65(1), 45–74. https://doi.org/10.1086/320037
Biemer, P. P., & Lyberg, L. E. (2003). Introduction to survey quality. Wiley. https://doi.org/10.1002/0471458740
Bradburn, N. M., Rips, L. J., & Shevell, S. K. (1987). Answering autobiographical questions: The impact of memory and inference on surveys. Science, 236(4798), 157–161. https://doi.org/10.1126/science.3563494
Coughlin, S. S. (1990). Recall bias in epidemiologic studies. Journal of Clinical Epidemiology, 43(1), 87–91. https://doi.org/10.1016/0895-4356(90)90060-3
Gaskell, G. D., Wright, D. B., & O'Muircheartaigh, C. A. (2000). Telescoping of landmark events: Implications for survey research. Public Opinion Quarterly, 64(1), 77–89. https://doi.org/10.1086/316761
Huttenlocher, J., Hedges, L. V., & Bradburn, N. M. (1990). Reports of elapsed time: Bounding and rounding processes in estimation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16(2), 196–213. https://doi.org/10.1037/0278-7393.16.2.196
Kahneman, D., Krueger, A. B., Schkade, D. A., Schwarz, N., & Stone, A. A. (2004). A survey method for characterizing daily life experience: The day reconstruction method. Science, 306(5702), 1776–1780. https://doi.org/10.1126/science.1103572
Loftus, E. F., & Marburger, W. (1983). Since the eruption of Mt. St. Helens, has anyone beaten you up? Improving the accuracy of retrospective reports with landmark events. Memory & Cognition, 11(2), 114–120. https://doi.org/10.3758/BF03213465
Means, B., & Loftus, E. F. (1991). When personal history repeats itself: Decomposing memories for recurring events. Applied Cognitive Psychology, 5(4), 297–318. https://doi.org/10.1002/acp.2350050402
Neter, J., & Waksberg, J. (1964). A study of response errors in expenditures data from household interviews. Journal of the American Statistical Association, 59(305), 18–55. https://doi.org/10.1080/01621459.1964.10480699
Raphael, K. (1987). Recall bias: A proposal for assessment and control. International Journal of Epidemiology, 16(2), 167–170. https://doi.org/10.1093/ije/16.2.167
Ross, M. (1989). Relation of implicit theories to the construction of personal histories. Psychological Review, 96(2), 341–357. https://doi.org/10.1037/0033-295X.96.2.341
Rubin, D. C., & Baddeley, A. D. (1989). Telescoping is not time compression: A model of the dating of autobiographical events. Memory & Cognition, 17(6), 653–661. https://doi.org/10.3758/BF03202626
Schacter, D. L. (1999). The seven sins of memory: Insights from psychology and cognitive neuroscience. American Psychologist, 54(3), 182–203. https://doi.org/10.1037/0003-066X.54.3.182
Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological momentary assessment. Annual Review of Clinical Psychology, 4, 1–32. https://doi.org/10.1146/annurev.clinpsy.3.022806.091415
Sudman, S., & Bradburn, N. M. (1973). Effects of time and memory factors on response in surveys. Journal of the American Statistical Association, 68(344), 805–815. https://doi.org/10.1080/01621459.1973.10481428
Thompson, C. P., Skowronski, J. J., Larsen, S. F., & Betz, A. L. (1996). Autobiographical memory: Remembering what and remembering when. Erlbaum.
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press. https://doi.org/10.1017/CBO9780511819322