Skip to content

Insights · Generating insight

The Role of Surprise in Discovery and Analytical Learning

Why surprise signals that our understanding is incomplete, what research in psychology, neuroscience and the history of science shows about how surprise drives learning, why people and organisations so often explain surprises away, and how analysts can use them deliberately.

ModQu research article · 1,536 words · 16 references (APA 7th edition)

Download PDF

A surprise is a moment when the world does not behave as we expected. In analysis, it might be a customer segment that buys more after a price rise, a campaign that works in the region where it was expected to fail, or a survey result that contradicts everything the team believed. Such moments are uncomfortable, and the instinct is often to check for errors and move on. Yet surprise is one of the most valuable signals an analyst can receive. It shows, precisely, where existing understanding is wrong or incomplete. This article explains why surprise matters for learning and discovery, why people and organisations so often waste it, and how to use it well.

Surprise as a learning signal

Theories of learning place surprise at their centre. Rescorla and Wagner (1972) proposed an influential model of how animals learn associations between events, in which learning occurs in proportion to the difference between what was expected and what actually happened. When an outcome is fully predicted, nothing new is learned; when it is unexpected, expectations are revised. Schultz et al. (1997) found that dopamine neurons in the brains of monkeys responded to unexpected rewards, but not to rewards that had become predictable, and responded with a dip in activity when an expected reward failed to appear. This "prediction error" signal appears to be a basic mechanism by which brains learn from experience.

Surprise also directs attention. Itti and Baldi (2009) defined surprise mathematically as the degree to which new data change an observer's beliefs, and found that people's eye movements while watching video were drawn to the locations that were most surprising in this sense. Meyer et al. (1997) described surprise as an emotional response to events that conflict with expectations, which interrupts ongoing activity, focuses attention on the unexpected event and prompts an effort to understand it.

That effort is important. Weiner (1985) reviewed research showing that people engage in spontaneous causal thinking, searching for explanations, mainly when events are unexpected or negative. Kahneman and Miller (1986) argued that surprising events evoke comparisons with what would normally have happened, which helps people identify what was unusual about them. Surprise, in other words, is what prompts people to ask "why?". Loewenstein (1994) proposed that curiosity arises from a perceived gap in knowledge, and surprise is one of the most direct ways in which such gaps become visible.

Surprise in scientific discovery

The history of science contains many discoveries that began with a surprise. Fleming (1929) reported that a mould contaminating one of Fleming's bacterial cultures had killed the surrounding bacteria, an unexpected observation that led to the discovery of penicillin. Merton and Barber (2004) traced the history of the concept of serendipity, and described how the "serendipity pattern" in science involves observing an unanticipated, anomalous and strategic finding that becomes the occasion for developing a new theory or extending an existing one. The important point is that the unexpected observation was noticed and taken seriously, not dismissed as an error.

Kuhn (1962) argued that scientific revolutions begin with anomalies, observations that do not fit the prevailing framework. At first, scientists try to explain them within the existing framework or set them aside. Only when anomalies accumulate and resist explanation does the framework itself come under question.

Dunbar (1997) studied scientists at work in several molecular biology laboratories, attending their meetings and recording their reasoning. Dunbar found that a large proportion of the findings scientists obtained were unexpected, and that how they dealt with these unexpected findings was central to their discoveries. A single unexpected result was usually attributed to a methodological error, but when similar unexpected results recurred, scientists began to look for new explanations, often by reasoning from analogies with other experiments.

How surprises are explained away

People do not always respond to surprise by learning. Chinn and Brewer (1993) reviewed how people respond to anomalous data that contradict their beliefs, and identified seven responses: ignoring the data, rejecting them, excluding them as irrelevant, holding them in abeyance for later, reinterpreting them to fit existing beliefs, making a peripheral change to the theory, or changing the theory itself. Only the last two involve real learning, and the first five are common.

Diagram of seven responses to anomalous data, arranged from no learning to deep learning: ignore, reject, exclude as irrelevant, hold in abeyance, reinterpret to fit, make a peripheral change, and change the theory. The first five are labelled "explaining away" and the last two "learning".
Figure 1. Seven ways people respond to data that contradict their beliefs. Only the last two change understanding. Diagram by ModQu, based on Chinn and Brewer (1993).

Hindsight also erodes surprise. Fischhoff (1975) showed that once people know an outcome, they overestimate how predictable it was. A surprising result can seem obvious within days of being reported, so that its lesson is lost: the team feels it "knew all along" and does not revise its assumptions. Pezzo (2003) found that surprising outcomes can either reduce or increase hindsight bias, depending on whether people succeed in making sense of them. When people readily find an explanation, the outcome comes to seem inevitable; when they do not, the surprise persists.

Organisations face similar pressures. Weick and Sutcliffe (2007) studied organisations that operate reliably in hazardous conditions, such as aircraft carriers and nuclear power plants, and found that they pay close attention to small surprises and weak signals of failure, treating them as information about the system rather than as anomalies to be explained away. Most organisations, by contrast, normalise small surprises until they become large ones. Sitkin (1992) argued that small failures are valuable for organisational learning because they draw attention to problems while the stakes are low, and that organisations which experience only success become complacent and less able to adapt.

Using surprise deliberately

Several practices help analysts turn surprise into learning.

Write down expectations first. Surprise requires an expectation. Before looking at results, record what you expect to find, ideally with numbers. This makes surprises visible and protects against hindsight, because the original expectation is on record.

Check for errors, then take the result seriously. Many surprising results are errors, and checking data and methods is the right first step. But if the result survives checking, it should be treated as a finding, not discarded because it is inconvenient.

Look for patterns in surprises. As Dunbar (1997) found, a single anomaly may be noise, but recurring anomalies point to something real. Keeping a log of surprising findings across projects allows patterns to emerge.

Ask what would have to be true. For each surprise, ask what would have to be true about customers, markets or the business for the result to make sense. This generates hypotheses that can be tested.

Share surprises. Organisations learn from surprises only if they are reported. Insight teams can make surprising findings a regular part of their reporting, so that decision makers see where the evidence challenges current assumptions.

Surprise in routine data

Surprise is not only a feature of one-off research projects. Organisations collect streams of routine data, such as daily sales, weekly complaints and monthly satisfaction scores, in which unexpected movements occur constantly. Most are noise. The analytical challenge is to separate the movements that signal something real from the ordinary variation that does not.

A practical approach is to define, for each important measure, what range of values would be unremarkable given its normal variation, seasonality and trend. Values outside that range are flagged as surprising and investigated; values inside it are not. This disciplines attention in two directions. It prevents teams from reacting to every small fluctuation as though it were news, and it ensures that genuinely unusual values are noticed rather than lost in a dashboard of numbers.

The same logic applies to forecasts. When an organisation forecasts sales, demand or customer numbers, the forecast errors are a record of its surprises. Reviewing the largest errors regularly, and asking what the forecast failed to anticipate, turns routine forecasting into a source of learning about the business. Errors that are consistently in one direction suggest a bias in the organisation's assumptions; errors that cluster in particular products, regions or periods point to something those assumptions do not capture.

An example

Consider a hypothetical meal-kit company that expected a new low-price range to attract younger, budget-conscious customers. When the results came in, the team was surprised: the new range sold best among older, higher-income households. The first reaction was to suspect an error in the customer data. After checking, the result held.

Asking what would have to be true for the result to make sense, the team developed a hypothesis: the low-price range also had simpler recipes and smaller portions, which might appeal to older couples whose children had left home. Interviews confirmed this. The company created a range aimed explicitly at smaller households, which became one of its most successful lines. The original expectation had been wrong, and the surprise showed exactly how.

Key takeaways

Surprise is the experience of a gap between expectation and reality, and it is a powerful signal for learning. Theories of learning, research on attention and the history of science all show that the unexpected is where new understanding begins. But people and organisations often explain surprises away, ignoring or reinterpreting them, and hindsight makes them seem obvious after the fact. Writing down expectations, taking surviving surprises seriously, looking for patterns and sharing unexpected findings help analysts turn surprise into discovery.

References

Chinn, C. A., & Brewer, W. F. (1993). The role of anomalous data in knowledge acquisition: A theoretical framework and implications for science instruction. Review of Educational Research, 63(1), 1–49. https://doi.org/10.3102/00346543063001001

Dunbar, K. (1997). How scientists think: On-line creativity and conceptual change in science. In T. B. Ward, S. M. Smith, & J. Vaid (Eds.), Creative thought: An investigation of conceptual structures and processes (pp. 461–493). American Psychological Association.

Fischhoff, B. (1975). Hindsight ≠ foresight: The effect of outcome knowledge on judgment under uncertainty. Journal of Experimental Psychology: Human Perception and Performance, 1(3), 288–299. https://doi.org/10.1037/0096-1523.1.3.288

Fleming, A. (1929). On the antibacterial action of cultures of a penicillium, with special reference to their use in the isolation of B. influenzæ. British Journal of Experimental Pathology, 10(3), 226–236.

Itti, L., & Baldi, P. (2009). Bayesian surprise attracts human attention. Vision Research, 49(10), 1295–1306. https://doi.org/10.1016/j.visres.2008.09.007

Kahneman, D., & Miller, D. T. (1986). Norm theory: Comparing reality to its alternatives. Psychological Review, 93(2), 136–153. https://doi.org/10.1037/0033-295X.93.2.136

Kuhn, T. S. (1962). The structure of scientific revolutions. University of Chicago Press.

Loewenstein, G. (1994). The psychology of curiosity: A review and reinterpretation. Psychological Bulletin, 116(1), 75–98. https://doi.org/10.1037/0033-2909.116.1.75

Merton, R. K., & Barber, E. (2004). The travels and adventures of serendipity: A study in sociological semantics and the sociology of science. Princeton University Press.

Meyer, W.-U., Reisenzein, R., & Schützwohl, A. (1997). Toward a process analysis of emotions: The case of surprise. Motivation and Emotion, 21(3), 251–274. https://doi.org/10.1023/A:1024422330338

Pezzo, M. V. (2003). Surprise, defence, or making sense: What removes hindsight bias? Memory, 11(4–5), 421–441. https://doi.org/10.1080/09658210244000603

Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: Current research and theory (pp. 64–99). Appleton-Century-Crofts.

Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593–1599. https://doi.org/10.1126/science.275.5306.1593

Sitkin, S. B. (1992). Learning through failure: The strategy of small losses. Research in Organizational Behavior, 14, 231–266.

Weick, K. E., & Sutcliffe, K. M. (2007). Managing the unexpected: Resilient performance in an age of uncertainty (2nd ed.). Jossey-Bass.

Weiner, B. (1985). "Spontaneous" causal thinking. Psychological Bulletin, 97(1), 74–84. https://doi.org/10.1037/0033-2909.97.1.74