Skip to content

Research · Research design

Natural Experiments in Business and Social Research

What natural experiments are, where they come from (lotteries, rules, borders, timing and shocks), famous examples from public health, economics and business, how to judge whether assignment really was as good as random, and how organisations can find natural experiments in their own data.

ModQu research article · 1,559 words · 15 references (APA 7th edition)

Download PDF

Some of the most convincing evidence about cause and effect comes from experiments that no researcher designed. A lottery decides who is called up for military service. A policy changes on one side of a border but not the other. A new technology reaches some towns months before others. In each case, events outside the researcher's control have divided people or places into groups in a way that is unrelated, or nearly unrelated, to the outcomes being studied. These situations are called natural experiments. This article explains what makes a natural experiment, where they come from, how to judge them, and how businesses can use the same logic.

What makes an experiment "natural"

In a randomised experiment, the researcher uses chance to decide who receives a treatment, which makes the treated and untreated groups alike in every respect except the treatment. In a natural experiment, the assignment is made by nature, by an institution or by circumstance, and the researcher argues that it was as good as random: unrelated to the characteristics of the units that might otherwise affect the outcome (Dunning, 2012).

Meyer (1995) called the best of these situations ones in which an exogenous event, such as a policy change, shifts treatment for some units and not others. This distinguishes a natural experiment from an ordinary observational comparison. Comparing people who chose to take a training course with people who did not is not a natural experiment, because the choice reflects motivation, ability and circumstances that also affect earnings. Comparing people who were admitted to an oversubscribed course by lottery with those who were not is a natural experiment, because the lottery, not the person, decided.

Dunning (2012) emphasises that the strength of a natural experiment lies in the design, not in statistical modelling. When assignment is as good as random, a simple comparison of means between groups can be credible. When it is not, no amount of statistical adjustment fully rescues the study.

A classic example

The best-known early natural experiment comes from John Snow's investigation of cholera in London. In the 1850s, two water companies supplied houses in the same districts of south London, with pipes from both companies running down the same streets. One company drew water from a stretch of the Thames contaminated with sewage; the other had moved its intake upstream. Rich and poor houses were served by both. Snow (1855) found that cholera deaths were far higher among households supplied by the contaminated company. Because the two groups were otherwise similar, the comparison provided strong evidence that cholera was spread through water, decades before the bacterium was identified.

Where natural experiments come from

Natural experiments arise from several kinds of event. Each produces a different kind of comparison and rests on a different assumption.

Table of five sources of natural experiments (lotteries, rules and thresholds, borders, timing and rollouts, shocks), each with a published example and the assumption the comparison relies on.
Figure 1. Five common sources of natural experiments, with an example of each and the assumption on which the comparison depends. Examples from Angrist (1990), Angrist and Krueger (1991), Card and Krueger (1994), Jensen (2007) and Card (1990); table by ModQu.

Lotteries

When an institution allocates something by lottery, it creates a true random assignment that a researcher can exploit later. Angrist (1990) studied the United States draft lotteries of the Vietnam War era, in which birth dates were drawn at random to decide the order in which men were called for service. Because birth dates are unrelated to earning potential, comparing men with high and low lottery numbers showed the effect of military service on later earnings, which turned out to be negative for white veterans.

Rules and thresholds

Many institutions apply rules that create sharp differences between otherwise similar people. Angrist and Krueger (1991) used the fact that compulsory schooling laws allowed students to leave school at a fixed age. Because students born early in the year reached that age after fewer years in school, quarter of birth slightly affected how much education people received, and this could be used to estimate the effect of schooling on earnings. Threshold rules of this kind are the basis of regression discontinuity designs.

Geography and borders

Policies often stop at a border, while the people and economies on either side remain similar. Card and Krueger (1994) compared fast-food restaurants in New Jersey, which raised its minimum wage, with those in neighbouring Pennsylvania, which did not. The comparison relies on the two areas following similar trends in the absence of the policy.

Timing and rollouts

When a technology, service or policy reaches different places at different times for reasons unrelated to local conditions, the staggered rollout creates natural comparison groups. Jensen (2007) studied the arrival of mobile phone coverage along the coast of Kerala, India, in the late 1990s. As coverage reached each region, fishermen could call markets while still at sea and sell where prices were best. Price differences between markets fell sharply and waste declined, region by region, as coverage arrived.

Shocks

Sudden, unexpected events can also create comparisons. Card (1990) studied the Mariel boatlift of 1980, when about 125,000 Cuban migrants arrived in Miami within a few months, expanding the city's labour force by around 7%. He found little effect on existing workers' wages or unemployment, compared with similar cities. Costello et al. (2003) studied a rural community in North Carolina in which a casino opened on a reservation and began paying a share of profits to every Eastern Cherokee family. Because the payments did not depend on family behaviour, they showed how moving out of poverty affected children's mental health.

Natural experiments in business

Businesses generate natural experiments constantly, often without noticing. A system outage disables a recommendation engine for half a day. A supplier problem leaves some stores without a product for a month. A competitor enters some markets and not others. A tax change alters prices in one jurisdiction. Einav et al. (2014) used listing data from eBay to study how sales taxes affect online shopping: because buyers in the seller's own state paid sales tax while other buyers did not, the same item listing faced different effective prices for different buyers.

Organisations can look for such events deliberately, asking whether anything changed suddenly for some customers, stores or regions and not others, and whether the reason for the change was unrelated to the outcome of interest. A planned rollout that proceeds by alphabetical order of store code, or by the order in which installation teams become available, may be close to random. A rollout that starts with the best-performing stores is not.

Judging a natural experiment

The central question is always whether assignment really was as good as random. Sekhon and Titiunik (2012) warned that many studies described as natural experiments are neither natural nor experiments: the assignment may have been influenced by the units themselves, or the treated and untreated groups may not be comparable even if assignment was random. Several checks help.

Rosenzweig and Wolpin (2000) added that natural experiments still require assumptions, often unstated, about how people respond. A natural experiment can establish an effect convincingly without explaining why it occurred.

Strengths and limits

Natural experiments combine two advantages that are rarely found together: credible causal comparison and real behaviour in real settings, often at a scale no researcher could fund. Craig et al. (2017) describe them as an essential tool in public health, where many important interventions, such as taxes, bans and planning decisions, cannot be randomised.

Their limits follow from the fact that the researcher cannot choose the treatment. Researchers must study the events that happen to occur, which may not match the question a decision-maker most needs answered. The groups affected may be unusual, the samples may be small, and the event may be unique. Angrist and Pischke (2009) argue that a good natural experiment is often worth more than a large but poorly identified dataset, but it is still one study in one setting, and its findings gain strength when similar events elsewhere produce similar results.

Conclusion

Natural experiments allow researchers to study causes that cannot be manipulated, by finding situations in which the world has done the randomising. Their value depends entirely on how close the assignment came to random, which must be argued from knowledge of the specific case and checked with data. For organisations, the practical lesson is to notice the sudden, uneven changes that happen in the course of business and ask whether they offer a comparison that a planned study could not.

References

Angrist, J. D. (1990). Lifetime earnings and the Vietnam era draft lottery: Evidence from Social Security administrative records. American Economic Review, 80(3), 313–336.

Angrist, J. D., & Krueger, A. B. (1991). Does compulsory school attendance affect schooling and earnings? Quarterly Journal of Economics, 106(4), 979–1014. https://doi.org/10.2307/2937954

Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press.

Card, D. (1990). The impact of the Mariel boatlift on the Miami labor market. Industrial and Labor Relations Review, 43(2), 245–257. https://doi.org/10.1177/001979399004300205

Card, D., & Krueger, A. B. (1994). Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania. American Economic Review, 84(4), 772–793.

Costello, E. J., Compton, S. N., Keeler, G., & Angold, A. (2003). Relationships between poverty and psychopathology: A natural experiment. JAMA, 290(15), 2023–2029. https://doi.org/10.1001/jama.290.15.2023

Craig, P., Katikireddi, S. V., Leyland, A., & Popham, F. (2017). Natural experiments: An overview of methods, approaches, and contributions to public health intervention research. Annual Review of Public Health, 38, 39–56. https://doi.org/10.1146/annurev-publhealth-031816-044327

Dunning, T. (2012). Natural experiments in the social sciences: A design-based approach. Cambridge University Press. https://doi.org/10.1017/CBO9781139084444

Einav, L., Knoepfle, D., Levin, J., & Sundaresan, N. (2014). Sales taxes and internet commerce. American Economic Review, 104(1), 1–26. https://doi.org/10.1257/aer.104.1.1

Imbens, G. W., & Angrist, J. D. (1994). Identification and estimation of local average treatment effects. Econometrica, 62(2), 467–475. https://doi.org/10.2307/2951620

Jensen, R. (2007). The digital provide: Information (technology), market performance, and welfare in the South Indian fisheries sector. Quarterly Journal of Economics, 122(3), 879–924. https://doi.org/10.1162/qjec.122.3.879

Meyer, B. D. (1995). Natural and quasi-experiments in economics. Journal of Business & Economic Statistics, 13(2), 151–161. https://doi.org/10.1080/07350015.1995.10524589

Rosenzweig, M. R., & Wolpin, K. I. (2000). Natural "natural experiments" in economics. Journal of Economic Literature, 38(4), 827–874. https://doi.org/10.1257/jel.38.4.827

Sekhon, J. S., & Titiunik, R. (2012). When natural experiments are neither natural nor experiments. American Political Science Review, 106(1), 35–57. https://doi.org/10.1017/S0003055411000542

Snow, J. (1855). On the mode of communication of cholera (2nd ed.). John Churchill.