The Generation of Knowledge · Knowledge production
Serendipity in the Generation of Knowledge
What serendipity means, famous cases from penicillin to the cosmic microwave background, why chance discoveries depend on prepared minds and supportive organisations, how serendipity has been classified and studied, and how research teams can make room for the unexpected without fooling themselves.
Many important discoveries were not the ones their discoverers were looking for. Penicillin, X-rays, the cosmic microwave background radiation and several of the first drugs for mental illness were all found while researchers were doing something else. The word for this kind of discovery is serendipity. It is often used loosely to mean good luck, but those who have studied it argue that it is better understood as an interaction between chance and skill: something unexpected happens, and someone has the knowledge, attention and freedom to recognise its value and follow it up. This article explains the origins of the idea, examines famous cases, reviews research on how serendipity works, and considers how organisations that generate knowledge can make room for it while guarding against the risk of seeing patterns in noise.
The origin of a word
The English writer Horace Walpole coined the word in a letter of 1754. He took it from a Persian fairy tale, The Three Princes of Serendip (Serendip being an old name for Sri Lanka), whose heroes, he wrote, were always making discoveries, by accidents and sagacity, of things they were not in quest of. Walpole's definition had two elements that remain central: accident, meaning that the discovery was not sought, and sagacity, meaning that the discoverer had the wit to see what the accident revealed.
The word was little used for nearly two centuries. Its history was traced in detail by the sociologist Robert Merton and his colleague Elinor Barber in a study written in the 1950s and finally published in English much later (Merton & Barber, 2004). They followed the word from Walpole's letter through its adoption by book collectors, scientists and eventually the general public, and showed how its meaning shifted along the way, sometimes emphasising luck and sometimes emphasising the sagacity of the discoverer.
Merton's serendipity pattern
Merton (1948) brought the concept into the study of research. He described a serendipity pattern in empirical research: the fairly common experience of observing an unanticipated, anomalous and strategic datum that becomes the occasion for developing a new theory or extending an existing one. Each of the three adjectives matters.
The datum is unanticipated: the research was designed to test one hypothesis, and this observation arose as a by-product. It is anomalous: it does not fit established theory or common sense, and so it demands explanation. And it is strategic: it has implications that bear on generalised theory, so that explaining it leads somewhere important. Merton argued that empirical research does not merely test theory; it can also initiate, reformulate and clarify theory, and the serendipity pattern is one of the main ways it does so.
Merton's account connects serendipity to a larger theme in the history of science. Kuhn (1962) argued that discovery commences with the awareness of anomaly, the recognition that nature has somehow violated the expectations set by the prevailing framework. Serendipitous discoveries are often anomalies that someone took seriously.
Famous cases
Penicillin
The best-known example is Alexander Fleming's observation of penicillin in 1928. Returning to his laboratory at St Mary's Hospital in London after a holiday, Fleming noticed that a culture plate of Staphylococcus bacteria had been contaminated by a mould, and that the bacteria near the mould had been destroyed. He grew the mould, identified it as a species of Penicillium, and showed that a substance it produced killed many kinds of bacteria (Fleming, 1929). The chance element was the contamination; the sagacity was Fleming's recognition that a spoiled plate showed something worth studying. Fleming himself suggested in his paper that the substance might be useful in the laboratory to isolate certain bacteria, and possibly as an antiseptic. Turning penicillin into a medicine required a further decade of work by other researchers, led by Howard Florey and Ernst Chain at Oxford, who developed methods to purify and produce it.
The cosmic microwave background
In 1964 the radio astronomers Arno Penzias and Robert Wilson, working at Bell Laboratories in New Jersey, were trying to use a sensitive antenna for radio astronomy. They found a persistent background noise that they could not eliminate, which was the same in every direction. After ruling out equipment faults, including droppings left by pigeons nesting in the antenna, they reported a measurement of "excess antenna temperature" (Penzias & Wilson, 1965). Physicists at nearby Princeton University recognised it as the faint radiation left over from the early universe, which their theory predicted. The discovery provided powerful evidence for the Big Bang model and earned Penzias and Wilson a Nobel Prize.
Drugs for mental illness
Serendipity has played a particularly large role in pharmacology. Ban (2006) reviewed the origins of the first modern drugs for psychiatric disorders and found that many were discovered by chance. Chlorpromazine, the first widely used antipsychotic, came from research on antihistamines used in surgery; the antidepressant effect of imipramine was noticed while it was being tested as a possible treatment for psychosis; and the use of lithium for mania followed from experiments that had a different aim. In each case, clinicians noticed effects that they had not been looking for and had the knowledge to see that those effects mattered.
Serendipity lost
Not every opportunity is taken. Barber and Fox (1958) studied a revealing pair of cases. Two medical scientists, working independently, each injected rabbits with the enzyme papain and observed that the rabbits' ears collapsed. One of them did not pursue the observation. The other, Lewis Thomas, did not pursue it at first either, but returned to it years later when he had time and a suitable method, and went on to make findings about cartilage and connective tissue. Barber and Fox called the first case "serendipity lost" and the second "serendipity gained", and showed that the difference lay not in the observation but in the circumstances of the scientists: their research priorities, the time and resources available to them, and their willingness to follow a puzzle that lay outside their current projects.
Chance favours the prepared mind
The most famous remark about serendipity is Louis Pasteur's, from a lecture of 1854: in the field of observation, chance favours only the prepared mind. Researchers who have studied serendipity agree that the "prepared mind" is the decisive ingredient. Fleming had spent years studying antibacterial substances and was therefore primed to see the significance of a cleared zone around a mould. Penzias and Wilson's careful elimination of instrumental causes was what made their measurement credible.
The neurologist James Austin offered a useful classification of the kinds of chance involved in discovery (Austin, 1978). Chance I is blind luck that comes to anyone. Chance II is luck that comes from motion: by being active, exploring widely and trying many things, one increases the number of fortunate encounters. Chance III is the luck of the prepared mind, in which background knowledge allows a person to recognise the significance of an accident. Chance IV is luck arising from a person's distinctive interests, habits and history, which lead them to unusual places and connections. On this view, much of what is called luck is in fact the result of activity, knowledge and individuality.
Roberts (1989), a chemist who collected many examples of accidental discoveries, distinguished serendipity, the accidental discovery of things not sought, from pseudoserendipity, the accidental discovery of ways to achieve an end that was sought. The vulcanisation of rubber, which Charles Goodyear had been trying to achieve for years before an accident showed him how, would count as pseudoserendipity. The distinction draws attention to the many cases where chance helps solve a problem that the discoverer already had.
Why unexpected results are often missed
If serendipity requires noticing the unexpected, it is limited by the ways human attention and reasoning work. Simons and Chabris (1999) showed that around half of the observers asked to count passes in a basketball video failed to notice a person in a gorilla suit walking through the scene. When attention is focused on a task, unexpected events can go entirely unseen.
Even when unexpected results are seen, they may be dismissed. Nickerson (1998) reviewed extensive evidence of confirmation bias, the tendency to seek, interpret and remember evidence in ways that support existing beliefs. A result that does not fit is easy to explain away as error, noise or contamination. Often that explanation is correct; most anomalies are indeed errors. The skill lies in knowing when to look more closely.
Dunbar and Fugelsang (2005) studied how scientists in molecular biology laboratories actually dealt with unexpected findings, by observing their laboratory meetings. They found that unexpected results were common, and that scientists' first response was usually to attribute them to problems with method. Only when an unexpected finding recurred did they begin to treat it as a real phenomenon, often by reasoning with analogies and discussing it with colleagues who brought different knowledge to the problem. Group discussion played an important role in turning surprises into discoveries.
Classifying serendipity
Recent research has tried to make the concept more precise. Yaqub (2018) studied hundreds of cases from the archives of Robert Merton and proposed a taxonomy of four types. In Walpolian serendipity, a targeted search leads to the discovery of something not sought. In Mertonian serendipity, a targeted search solves the intended problem, but by an unexpected route. In Bushian serendipity, named after the advocate of curiosity-driven research Vannevar Bush, an untargeted search leads to the solution of an immediate problem. In Stephanian serendipity, untargeted exploration yields something that later solves a problem no one had yet posed. Yaqub also identified mechanisms through which serendipity arises, including theory-led insight, the observations of a perceptive individual, errors and accidents, and connections that emerge from networks of people.
Philosophers have asked whether serendipity is compatible with a rational account of science. Kantorovich (1993) argued that serendipity and tinkering are a normal and important source of novelty in science, with creative discovery playing a role similar to that of blind mutation in biological evolution: it supplies new possibilities that are then selected by testing. Copeland (2019) argued that serendipity should be understood as discovery at the intersection of chance and the wisdom of the discoverer, and that its frequency in the history of science says something important about how knowledge grows: researchers cannot plan every advance, but they can prepare themselves to exploit chances when they arise.
Serendipity in information seeking
Serendipity is not confined to laboratories. Researchers in information science have studied how people come across useful information they were not looking for. Foster and Ford (2003) interviewed interdisciplinary researchers and found that serendipitous encounters with information were an important part of how they found ideas, and that researchers sometimes deliberately placed themselves where such encounters were likely, for example by browsing library shelves or reading widely. Makri and Blandford (2012) developed a process model from interviews with researchers and other professionals, in which serendipity involves making a new connection, recognising its potential value, and following it up to produce a valuable outcome.
McCay-Peet and Toms (2015) studied serendipity in the workplace and described it as a process involving a trigger, a connection, a follow-up and a valuable outcome. They found that both the person's traits, such as curiosity and openness, and the environment, such as access to diverse information and people, influenced how often serendipity occurred. Rubin et al. (2011) analysed personal accounts of chance encounters and identified the prepared mind, the act of noticing and the element of chance as recurring facets.
Serendipity matters in business too. Dew (2009) argued that serendipity plays an important role in entrepreneurship, when prior knowledge, active search and chance events combine to reveal opportunities. In qualitative social research, Fine and Deegan (1996) described how chance encounters, unplanned conversations and unexpected events during fieldwork often produce the most revealing insights.
Can serendipity be managed?
If serendipity depends partly on chance, it cannot be planned. But the research reviewed above suggests that its likelihood can be increased, and that organisations can either encourage or suppress it. de Rond (2014) argued that serendipity is better seen as a capability than as an event: the ability to notice and make use of unexpected connections can be developed. Busch (2020) drew on research and interviews to recommend practical habits, such as setting aside time for exploration, connecting with people from different fields and treating surprises as possible opportunities rather than as interruptions.
Several organisational conditions recur in the literature:
Slack and freedom. The case of the floppy-eared rabbits shows that following an unexpected observation requires time and resources outside the main project (Barber & Fox, 1958). Organisations that schedule every hour and tie all effort to predefined deliverables leave no room for this.
Diverse contacts. Dunbar and Fugelsang (2005) found that colleagues with different backgrounds helped turn surprising results into discoveries. Networks that cross departments and disciplines make useful connections more likely.
Exploration as well as confirmation. Tukey (1977) urged analysts to explore data openly before testing pre-specified hypotheses. Exploration is where unexpected patterns are most likely to appear.
A culture that welcomes surprises. If unexpected results are treated as embarrassing failures, they will be hidden. If they are discussed openly, they can be investigated.
Support for curiosity-driven work. Bush (1945) argued that basic research performed without thought of practical ends was the source of the new knowledge on which practical progress depends. Yaqub's (2018) Bushian and Stephanian types of serendipity, in which untargeted exploration later proves useful, depend on this kind of freedom.
The danger of false serendipity
The openness that makes serendipity possible also creates a danger. Data sets of any size contain many patterns that arise by chance alone. An analyst who searches through them looking for something surprising will always find something, and much of it will be noise. If such patterns are then presented as though they had been predicted in advance, the result is misleading. Kerr (1998) called this practice HARKing, hypothesizing after the results are known, and showed how it distorts the scientific record by making chance findings look like confirmed predictions.
The solution is not to ignore unexpected findings but to treat them honestly. A surprising pattern found during exploration is a hypothesis, not a conclusion. It should be reported as exploratory, tested on new data where possible, and checked against other sources and plausible explanations, including errors in collection or processing. Many apparent discoveries dissolve under this scrutiny; those that survive are much more likely to be real. This is exactly the process that Penzias and Wilson followed when they spent months eliminating instrumental explanations for their excess noise before reporting it.
Serendipity in survey and market research
Applied research may seem an unlikely home for serendipity, since projects are usually commissioned to answer specific questions. But surveys and related studies generate many opportunities for unexpected discoveries.
Open-ended responses. Answers to open questions often raise issues that the researchers did not anticipate and that closed questions would never have revealed. Reading them carefully, rather than only coding them into predefined categories, is a common source of insight.
Unexpected subgroup differences. A result that holds for the population as a whole may differ sharply for a particular group. Such differences can point to new segments, needs or problems, but they need confirmation because, with many subgroups, some differences will appear by chance.
Pilot testing and fieldwork. Cognitive interviews and pilot studies often reveal that respondents understand a question differently from the way the researchers intended. These surprises improve the instrument and can also reveal how people actually think about the topic.
Data about the process itself. Information about how a survey was completed, such as where respondents stopped or which questions they skipped, can reveal unexpected problems or interests.
Connections across projects. An organisation that conducts many studies may notice patterns that no single study shows. Keeping data and findings accessible across projects creates the conditions for this kind of discovery.
In each case, the principles are the same: notice the unexpected, check whether it is real, and if it is, follow it up and report it honestly as a finding that was not anticipated.
Conclusion
Serendipity is not simply luck. It is a pattern in which chance provides an opportunity and prepared, curious and well-supported people make use of it. The history of science shows that many important discoveries arose in this way, and research on how serendipity works shows that it depends on attention, knowledge, time, diverse contacts and a culture that welcomes surprises. It also shows the need for discipline, since the search for unexpected patterns can easily produce false discoveries. For organisations that generate knowledge, the lesson is to design work that leaves room for the unexpected, to train people to notice and question surprises, and to test every surprising finding before treating it as knowledge.
References
Austin, J. H. (1978). Chase, chance, and creativity: The lucky art of novelty. Columbia University Press.
Ban, T. A. (2006). The role of serendipity in drug discovery. Dialogues in Clinical Neuroscience, 8(3), 335–344. https://doi.org/10.31887/DCNS.2006.8.3/tban
Barber, B., & Fox, R. C. (1958). The case of the floppy-eared rabbits: An instance of serendipity gained and serendipity lost. American Journal of Sociology, 64(2), 128–136. https://doi.org/10.1086/222420
Busch, C. (2020). The serendipity mindset: The art and science of creating good luck. Riverhead Books.
Bush, V. (1945). Science, the endless frontier: A report to the President. United States Government Printing Office.
Copeland, S. (2019). On serendipity in science: Discovery at the intersection of chance and wisdom. Synthese, 196(6), 2385–2406. https://doi.org/10.1007/s11229-017-1544-3
de Rond, M. (2014). The structure of serendipity. Culture and Organization, 20(5), 342–358. https://doi.org/10.1080/14759551.2014.967451
Dew, N. (2009). Serendipity in entrepreneurship. Organization Studies, 30(7), 735–753. https://doi.org/10.1177/0170840609104815
Dunbar, K., & Fugelsang, J. (2005). Causal thinking in science: How scientists and students interpret the unexpected. In M. E. Gorman, R. D. Tweney, D. C. Gooding, & A. P. Kincannon (Eds.), Scientific and technological thinking (pp. 57–79). Lawrence Erlbaum Associates.
Fine, G. A., & Deegan, J. G. (1996). Three principles of Serendip: Insight, chance, and discovery in qualitative research. International Journal of Qualitative Studies in Education, 9(4), 434–447. https://doi.org/10.1080/0951839960090405
Fleming, A. (1929). On the antibacterial action of cultures of a penicillium, with special reference to their use in the isolation of B. influenzæ. British Journal of Experimental Pathology, 10(3), 226–236.
Foster, A., & Ford, N. (2003). Serendipity and information seeking: An empirical study. Journal of Documentation, 59(3), 321–340. https://doi.org/10.1108/00220410310472518
Kantorovich, A. (1993). Scientific discovery: Logic and tinkering. State University of New York Press.
Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. https://doi.org/10.1207/s15327957pspr0203_4
Kuhn, T. S. (1962). The structure of scientific revolutions. University of Chicago Press.
Makri, S., & Blandford, A. (2012). Coming across information serendipitously – Part 1: A process model. Journal of Documentation, 68(5), 684–705. https://doi.org/10.1108/00220411211256030
McCay-Peet, L., & Toms, E. G. (2015). Investigating serendipity: How it unfolds and what may influence it. Journal of the Association for Information Science and Technology, 66(7), 1463–1476. https://doi.org/10.1002/asi.23273
Merton, R. K. (1948). The bearing of empirical research upon the development of social theory. American Sociological Review, 13(5), 505–515.
Merton, R. K., & Barber, E. (2004). The travels and adventures of serendipity: A study in sociological semantics and the sociology of science. Princeton University Press.
Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. https://doi.org/10.1037/1089-2680.2.2.175
Penzias, A. A., & Wilson, R. W. (1965). A measurement of excess antenna temperature at 4080 Mc/s. The Astrophysical Journal, 142, 419–421. https://doi.org/10.1086/148307
Roberts, R. M. (1989). Serendipity: Accidental discoveries in science. Wiley.
Rubin, V. L., Burkell, J., & Quan-Haase, A. (2011). Facets of serendipity in everyday chance encounters: A grounded theory approach to blog analysis. Information Research, 16(3), Article 488.
Simons, D. J., & Chabris, C. F. (1999). Gorillas in our midst: Sustained inattentional blindness for dynamic events. Perception, 28(9), 1059–1074. https://doi.org/10.1068/p281059
Tukey, J. W. (1977). Exploratory data analysis. Addison-Wesley.
Yaqub, O. (2018). Serendipity: Towards a taxonomy and a theory. Research Policy, 47(1), 169–179. https://doi.org/10.1016/j.respol.2017.10.007