Insights · Insight quality and pitfalls
The Role of Domain Expertise in Filtering and Validating Insights
How people who know a business, market or process well help separate real insights from artefacts and noise, when expert judgement can be trusted and when it misleads, and how to combine expertise with data.
Data analysis can produce many candidate insights. Some are real, some are artefacts of how data were recorded, and some are chance. Domain experts, people who know the market, the product or the process well, are often the fastest filter between the two. They can tell an analyst that a sudden rise in returns coincides with a change in the returns policy, or that a "segment" is simply the customers of one large distributor. But experts can also dismiss real findings because they conflict with what the experts already believe. This article explains what expertise contributes to validating insight and how to use it well.
What experts know that data do not
Experts see patterns that novices miss. Chase and Simon (1973) found that chess masters could recall positions from real games far better than weaker players, but had no such advantage for randomly arranged pieces. Their skill lay in recognising meaningful configurations built up through experience. Ericsson et al. (1993) argued that such expertise is built through years of deliberate practice with feedback.
In insight work, this knowledge is valuable in several ways.
Experts know what the data mean, including quirks that are not documented, such as a field that changed definition when a system was replaced. They know about events that can create apparent patterns, such as promotions, strikes, weather or a competitor's price change. They can judge whether a finding is plausible given how customers and the business behave, propose a mechanism that can be tested, and say whether anything practical could be done with the result. Domingos (2012) noted that in machine learning, much of the difference between successful and unsuccessful projects lies in the choice of features, and that this choice depends heavily on knowledge of the domain.
When expert judgement can be trusted
Expertise is not equally reliable everywhere. Kahneman and Klein (2009) concluded that intuitive expert judgement is likely to be accurate only when the environment is regular enough to be predictable and the expert has had sufficient opportunity to learn its regularities through practice and timely feedback. Shanteau (1992) observed that experts performed well in some fields, such as weather forecasting and chess, and poorly in others, such as clinical psychology and stock picking, and linked the difference to features of the task, including whether the subject matter changes and whether feedback is available.
Tetlock (2005) studied thousands of forecasts by political and economic experts over many years and found that they performed only slightly better than simple statistical rules, and that experts who organised their thinking around one big idea did worse than those who drew on many sources and adjusted their views. Research comparing expert judgement with statistical prediction, beginning with Meehl (1954), has repeatedly found that simple formulas match or beat expert judgement in many prediction tasks. A meta-analysis by Grove et al. (2000) found that mechanical prediction was on average about 10% more accurate than clinical judgement, and that experts outperformed formulas in only a small minority of studies.
The lesson is that experts are often better at explaining and checking data than at predicting outcomes without it.
When expertise gets in the way
Deep knowledge can also make people resistant to new information. Dane (2010) described cognitive entrenchment, a high level of stability in a person's way of thinking about a domain, which can make experts less able to adapt when the domain changes or when a problem falls outside their usual patterns. An expert who has seen a market behave one way for many years may reject evidence that it has changed. This is a particular risk for insights that challenge the organisation's existing strategy.
Combining expertise with data
The best results usually come from combining the two:
- Involve experts early. Ask them what the data mean and which events might distort them before analysis begins.
- Ask for mechanisms, not verdicts. When an expert doubts a finding, ask what would have to be true for it to be wrong, and check that.
- Separate plausibility from truth. A finding that surprises experts deserves more checking, not automatic rejection.
- Use several experts. People from sales, operations and finance notice different artefacts.
An example
Consider a hypothetical analysis at a food manufacturer that finds sales of one product line rose sharply in three regions after a packaging change. The analysts conclude that the new packaging works and propose rolling it out everywhere. A regional sales manager, shown the result, points out that the same three regions gained a new supermarket customer in the same month, which added shelf space for the whole range. The analysts then compare the product line with other lines sold to the new customer and find that most of the rise is explained by the new listings. A smaller effect of the packaging remains, and a controlled test in two further regions is planned to measure it properly. The expert did not decide the question, but supplied the fact that made the right check possible.
Key takeaways
Domain experts help validate insights by explaining what data mean, identifying events that create false patterns, judging plausibility and proposing testable mechanisms. Their intuition is reliable mainly in stable environments with good feedback, and simple statistical methods often predict better than unaided judgement. Expertise can also become entrenched and lead to the rejection of real but unwelcome findings. Combining expert checks with evidence, and asking experts for mechanisms rather than verdicts, gets the benefit of expertise while limiting its risks.
References
Chase, W. G., & Simon, H. A. (1973). Perception in chess. Cognitive Psychology, 4(1), 55–81. https://doi.org/10.1016/0010-0285(73)90004-2
Dane, E. (2010). Reconsidering the trade-off between expertise and flexibility: A cognitive entrenchment perspective. Academy of Management Review, 35(4), 579–603. https://doi.org/10.5465/amr.35.4.zok579
Domingos, P. (2012). A few useful things to know about machine learning. Communications of the ACM, 55(10), 78–87. https://doi.org/10.1145/2347736.2347755
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363–406. https://doi.org/10.1037/0033-295X.100.3.363
Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12(1), 19–30. https://doi.org/10.1037/1040-3590.12.1.19
Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. American Psychologist, 64(6), 515–526. https://doi.org/10.1037/a0016755
Meehl, P. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. University of Minnesota Press.
Shanteau, J. (1992). Competence in experts: The role of task characteristics. Organizational Behavior and Human Decision Processes, 53(2), 252–266. https://doi.org/10.1016/0749-5978(92)90064-E
Tetlock, P. E. (2005). Expert political judgment: How good is it? How can we know? Princeton University Press.