Skip to content

Insights · Insight quality and pitfalls

The Role of Domain Expertise in Filtering and Validating Insights

How people who know a business, market or process well help separate real insights from artefacts and noise, when expert judgement can be trusted and when it misleads, and how to combine expertise with data.

ModQu research article · 930 words · 9 references (APA 7th edition)

Download PDF

Data analysis can produce many candidate insights. Some are real, some are artefacts of how data were recorded, and some are chance. Domain experts, people who know the market, the product or the process well, are often the fastest filter between the two. They can tell an analyst that a sudden rise in returns coincides with a change in the returns policy, or that a "segment" is simply the customers of one large distributor. But experts can also dismiss real findings because they conflict with what the experts already believe. This article explains what expertise contributes to validating insight and how to use it well.

What experts know that data do not

Experts see patterns that novices miss. Chase and Simon (1973) found that chess masters could recall positions from real games far better than weaker players, but had no such advantage for randomly arranged pieces. Their skill lay in recognising meaningful configurations built up through experience. Ericsson et al. (1993) argued that such expertise is built through years of deliberate practice with feedback.

In insight work, this knowledge is valuable in several ways.

Table of five checks that domain experts are well placed to make on a candidate insight: data meaning (what does this field really record, and has that changed?), known events (did a policy, system, pricing or seasonal change occur at the time?), plausibility (does the finding fit how customers and the business actually behave?), mechanism (what would have to be true for this to be real, and can we check it?), and actionability (can anything practical be done with it?).
Figure 1. Five checks domain experts are well placed to make on a candidate insight. Diagram by ModQu.

Experts know what the data mean, including quirks that are not documented, such as a field that changed definition when a system was replaced. They know about events that can create apparent patterns, such as promotions, strikes, weather or a competitor's price change. They can judge whether a finding is plausible given how customers and the business behave, propose a mechanism that can be tested, and say whether anything practical could be done with the result. Domingos (2012) noted that in machine learning, much of the difference between successful and unsuccessful projects lies in the choice of features, and that this choice depends heavily on knowledge of the domain.

When expert judgement can be trusted

Expertise is not equally reliable everywhere. Kahneman and Klein (2009) concluded that intuitive expert judgement is likely to be accurate only when the environment is regular enough to be predictable and the expert has had sufficient opportunity to learn its regularities through practice and timely feedback. Shanteau (1992) observed that experts performed well in some fields, such as weather forecasting and chess, and poorly in others, such as clinical psychology and stock picking, and linked the difference to features of the task, including whether the subject matter changes and whether feedback is available.

Tetlock (2005) studied thousands of forecasts by political and economic experts over many years and found that they performed only slightly better than simple statistical rules, and that experts who organised their thinking around one big idea did worse than those who drew on many sources and adjusted their views. Research comparing expert judgement with statistical prediction, beginning with Meehl (1954), has repeatedly found that simple formulas match or beat expert judgement in many prediction tasks. A meta-analysis by Grove et al. (2000) found that mechanical prediction was on average about 10% more accurate than clinical judgement, and that experts outperformed formulas in only a small minority of studies.

The lesson is that experts are often better at explaining and checking data than at predicting outcomes without it.

When expertise gets in the way

Deep knowledge can also make people resistant to new information. Dane (2010) described cognitive entrenchment, a high level of stability in a person's way of thinking about a domain, which can make experts less able to adapt when the domain changes or when a problem falls outside their usual patterns. An expert who has seen a market behave one way for many years may reject evidence that it has changed. This is a particular risk for insights that challenge the organisation's existing strategy.

Combining expertise with data

The best results usually come from combining the two:

An example

Consider a hypothetical analysis at a food manufacturer that finds sales of one product line rose sharply in three regions after a packaging change. The analysts conclude that the new packaging works and propose rolling it out everywhere. A regional sales manager, shown the result, points out that the same three regions gained a new supermarket customer in the same month, which added shelf space for the whole range. The analysts then compare the product line with other lines sold to the new customer and find that most of the rise is explained by the new listings. A smaller effect of the packaging remains, and a controlled test in two further regions is planned to measure it properly. The expert did not decide the question, but supplied the fact that made the right check possible.

Key takeaways

Domain experts help validate insights by explaining what data mean, identifying events that create false patterns, judging plausibility and proposing testable mechanisms. Their intuition is reliable mainly in stable environments with good feedback, and simple statistical methods often predict better than unaided judgement. Expertise can also become entrenched and lead to the rejection of real but unwelcome findings. Combining expert checks with evidence, and asking experts for mechanisms rather than verdicts, gets the benefit of expertise while limiting its risks.

References

Chase, W. G., & Simon, H. A. (1973). Perception in chess. Cognitive Psychology, 4(1), 55–81. https://doi.org/10.1016/0010-0285(73)90004-2

Dane, E. (2010). Reconsidering the trade-off between expertise and flexibility: A cognitive entrenchment perspective. Academy of Management Review, 35(4), 579–603. https://doi.org/10.5465/amr.35.4.zok579

Domingos, P. (2012). A few useful things to know about machine learning. Communications of the ACM, 55(10), 78–87. https://doi.org/10.1145/2347736.2347755

Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363–406. https://doi.org/10.1037/0033-295X.100.3.363

Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12(1), 19–30. https://doi.org/10.1037/1040-3590.12.1.19

Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. American Psychologist, 64(6), 515–526. https://doi.org/10.1037/a0016755

Meehl, P. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. University of Minnesota Press.

Shanteau, J. (1992). Competence in experts: The role of task characteristics. Organizational Behavior and Human Decision Processes, 53(2), 252–266. https://doi.org/10.1016/0749-5978(92)90064-E

Tetlock, P. E. (2005). Expert political judgment: How good is it? How can we know? Princeton University Press.