Skip to content

Insights · Insight quality and pitfalls

Insight Decay: How Long Does an Insight Stay True?

Why insights lose their accuracy over time, how quickly different kinds of knowledge decay, how to detect when an insight is no longer true, and practical ways to date, review and refresh an organisation's stock of insight.

ModQu research article · 1,429 words · 15 references (APA 7th edition)

Download PDF

An insight that was true when it was discovered may not be true now. Customers change, competitors move, technology shifts and the organisation itself changes what it offers. Yet insights, once accepted, tend to live on in presentations, strategies and the beliefs of managers long after the evidence behind them has gone out of date. Insight decay is the gradual loss of accuracy of knowledge over time. Recognising it, and managing it, is an important part of keeping an organisation's understanding of its customers and markets current.

Knowledge has a half-life

The idea that knowledge decays is not new. Burton and Kebler (1960) borrowed the physicists' concept of half-life to describe how quickly the scientific literature in a field becomes obsolete, measured by how long it takes for half of the citations to a field's literature to refer to work older than a given age. They found that half-lives differed greatly between fields. Arbesman (2012) extended the idea to facts more generally, arguing that knowledge in many fields changes in measurable and somewhat predictable ways, with some kinds of facts overturned or refined much faster than others.

Even well-established findings can be reversed. Ioannidis (2005) examined highly cited clinical research studies and found that a substantial share were later contradicted or found to have exaggerated effects by subsequent research. Prasad et al. (2013) reviewed a decade of articles in a leading medical journal and identified 146 established medical practices that were found by later studies to be no better than, or worse than, the practices they had replaced. If knowledge about the human body decays in this way, knowledge about markets, which are shaped by fashion, technology and competition, can be expected to decay faster.

Why insights decay

Insights decay for several reasons.

The world changes. Customer preferences, technology, prices and competitors change. An insight about how customers research purchases may have been accurate before the spread of smartphones and inaccurate after. Golder and Tellis (2004) showed that products follow life cycles with sharp turning points, so that patterns of demand observed in a growth phase can change abruptly when a market matures.

The effects of actions fade. Dekimpe and Hanssens (1995) studied the long-term effects of marketing on sales and found that in many cases, the effects of marketing actions were temporary, with sales returning to their previous path. An insight about what drives sales may reflect a short-term response that does not persist.

The original finding was overstated. Some insights were never as strong as they appeared. Schooler (2011) discussed the "decline effect", the tendency for the size of some scientific findings to shrink in later studies, and argued that it was partly explained by unpublished results that would have shown smaller effects. The same pattern occurs in business when the first, most impressive analysis becomes the accepted figure.

Acting on an insight changes it. When an organisation acts on an insight, and competitors copy it, the conditions that made it true can disappear. An insight that a particular customer group is underserved stops being true once several companies target that group.

Models decay too

The same problem affects predictive models. In machine learning, "concept drift" refers to changes over time in the relationship between the inputs to a model and the outcome it predicts. Gama et al. (2014) reviewed methods for detecting and adapting to concept drift, and emphasised that models used in changing environments need continuous monitoring. Lazer et al. (2014) described how a widely publicised model for estimating influenza levels from search data became increasingly inaccurate, partly because changes to the search engine altered how people searched.

Organisations hold on to old insights

Organisations are often slow to recognise that their insights have decayed. Levinthal and March (1993) described the "myopia of learning", in which organisations favour the near term and the familiar, and learn in ways that reinforce what has worked before. Leonard-Barton (1992) showed how the capabilities that make an organisation successful can become "core rigidities" when circumstances change. Christensen (1997) described how successful firms, listening closely to existing customers, missed the shift to technologies that served different customers. In each case, knowledge that had been accurate continued to guide decisions after it had stopped being so.

Different insights decay at different rates

Not all insights decay at the same rate. Insights about basic human psychology, such as the tendency to weigh losses more heavily than gains (Kahneman & Tversky, 1979), change slowly. Insights about how a category is used, or about the needs of a customer group, change at a moderate pace. Insights about media habits, digital channels and specific tactics can become outdated within months.

Line chart showing, for three hypothetical types of insight, the share still accurate over ten years. Insights about basic human behaviour remain about 90% accurate after ten years. Insights about category needs and usage fall to about 50% after six years. Insights about channels and tactics fall to about 50% within two years and to about 10% after seven.
Figure 1. Different kinds of insight decay at different rates. Schematic illustration, chart by ModQu.

Signs that an insight has decayed

Decay is rarely announced. It usually shows up first as small inconsistencies that are easy to explain away. Several signs deserve attention.

Forecasts based on the insight start to miss. If a model or rule of thumb built on an insight produces larger errors than it used to, the relationship it describes may have changed.

Front-line staff tell a different story. Sales and service staff often notice changes in customer behaviour before formal research does. When their accounts no longer match the accepted insight, the insight should be checked.

New customer groups behave differently. An insight drawn from early customers may not hold for the customers who arrive later. Rogers (2003) described how the people who adopt an innovation late differ from early adopters in their motivations, resources and sources of information, so findings about early users can mislead once a product spreads.

The evidence is old or second-hand. An insight that is quoted often but whose original source no one can locate, or whose data were collected many years ago, is a candidate for review, however familiar it feels.

Exploiting old insights versus exploring for new ones

March (1991) described the tension in organisational learning between exploitation, using and refining what is already known, and exploration, searching for new knowledge. Exploitation produces reliable, near-term returns, so organisations tend to favour it. Exploration is uncertain and slow to pay off, but without it the stock of knowledge becomes outdated. Managing insight decay is partly a matter of protecting a share of research effort for questions whose answers the organisation believes it already knows.

An example

Consider a hypothetical retailer whose segmentation, built six years ago, found that younger customers rarely bought homeware online because they wanted to see and touch products first. Store layouts, advertising and the website were all designed around this insight. Over time, online homeware sales to younger customers grew, but the growth was treated as noise because it contradicted the accepted segmentation.

When the retailer finally repeated the research, it found that the original insight had reversed. Improved photography, free returns and customer reviews had removed most of the uncertainty that once kept younger customers in stores. The cost of the delay was several years of under-investment in the fastest-growing channel. A review date on the original insight, and a willingness to treat the unexpected online growth as a signal, would have prompted the check much earlier.

Managing insight decay

Several practices help.

Date every insight. Record when the evidence was collected, not only when the report was written, and show the date wherever the insight is used.

Assign a review date. Based on the type of insight, set a date by which it should be checked, sooner for fast-changing topics.

Track key assumptions. Identify the few insights on which major decisions depend, and monitor indicators that would show if they stopped being true.

Watch for surprises. Results that contradict an established insight may be early signs of decay. Treat them as prompts to re-examine the insight rather than as anomalies.

Monitor models. Compare predictive models' accuracy over time and retrain or rebuild them when performance declines.

Retire insights explicitly. When an insight is no longer supported, record that it has been retired and why, so that it does not resurface in later work.

Key takeaways

Insights lose accuracy over time because the world changes, the effects of actions fade, original findings were sometimes overstated, and acting on an insight can change the conditions that made it true. Predictive models suffer from the same problem. Organisations tend to hold on to old insights, especially those associated with past success. Different kinds of insight decay at different rates. Dating insights, setting review dates, tracking key assumptions, watching for surprises, monitoring models and retiring outdated insights explicitly help keep an organisation's knowledge current.

References

Arbesman, S. (2012). The half-life of facts: Why everything we know has an expiration date. Current.

Burton, R. E., & Kebler, R. W. (1960). The "half-life" of some scientific and technical literatures. American Documentation, 11(1), 18–22. https://doi.org/10.1002/asi.5090110105

Christensen, C. M. (1997). The innovator's dilemma: When new technologies cause great firms to fail. Harvard Business School Press.

Dekimpe, M. G., & Hanssens, D. M. (1995). The persistence of marketing effects on sales. Marketing Science, 14(1), 1–21. https://doi.org/10.1287/mksc.14.1.1

Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). A survey on concept drift adaptation. ACM Computing Surveys, 46(4), Article 44. https://doi.org/10.1145/2523813

Golder, P. N., & Tellis, G. J. (2004). Growing, growing, gone: Cascades, diffusion, and turning points in the product life cycle. Marketing Science, 23(2), 207–218. https://doi.org/10.1287/mksc.1040.0057

Ioannidis, J. P. A. (2005). Contradicted and initially stronger effects in highly cited clinical research. JAMA, 294(2), 218–228. https://doi.org/10.1001/jama.294.2.218

Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47(2), 263–291. https://doi.org/10.2307/1914185

Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014). The parable of Google Flu: Traps in big data analysis. Science, 343(6176), 1203–1205. https://doi.org/10.1126/science.1248506

Leonard-Barton, D. (1992). Core capabilities and core rigidities: A paradox in managing new product development. Strategic Management Journal, 13(S1), 111–125. https://doi.org/10.1002/smj.4250131009

Levinthal, D. A., & March, J. G. (1993). The myopia of learning. Strategic Management Journal, 14(S2), 95–112. https://doi.org/10.1002/smj.4250141009

March, J. G. (1991). Exploration and exploitation in organizational learning. Organization Science, 2(1), 71–87. https://doi.org/10.1287/orsc.2.1.71

Prasad, V., Vandross, A., Toomey, C., Cheung, M., Rho, J., Quinn, S., Chacko, S. J., Borkar, D., Gall, V., Selvaraj, S., Ho, N., & Cifu, A. (2013). A decade of reversal: An analysis of 146 contradicted medical practices. Mayo Clinic Proceedings, 88(8), 790–798. https://doi.org/10.1016/j.mayocp.2013.05.012

Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press.

Schooler, J. (2011). Unpublished results hide the decline effect. Nature, 470(7335), 437. https://doi.org/10.1038/470437a