Insights · Tools and the future
Decision-Support Systems Designed Around Insight Delivery
How to design decision-support systems that deliver insight at the point of decision, covering the history of the field, evidence on what makes such systems effective, the risks of automation bias and alert fatigue, and practical design principles.
Most organisations hold more data than their managers can use. Reports, dashboards and analytics tools are produced in large numbers, yet many decisions are still made without reference to them. A decision-support system is a computer-based system that helps people make decisions by bringing together data, models and an interface that lets the user explore options. Systems of this kind have been studied for more than fifty years. The lesson of that research is that the value of a decision-support system depends less on the sophistication of its analysis than on whether it delivers the right insight to the right person at the moment a decision is made. This article reviews what is known about designing systems around insight delivery.
Where decision support came from
Gorry and Scott Morton (1971) introduced the term "decision support systems" in a framework that combined two ideas. From Simon (1960), they took the distinction between structured decisions, which follow routine procedures, and unstructured decisions, which require judgement. From Anthony's classification of management activity, they took the distinction between operational control, management control and strategic planning. They argued that computers had so far been applied mainly to structured operational decisions, and that the larger opportunity lay in supporting semi-structured decisions, where a computer can handle part of the problem while a person supplies judgement for the rest.
Little (1970) set out a related idea from the perspective of management science. Little observed that managers rarely used the mathematical models built for them and argued that a model should be simple, robust, easy to control, adaptive and easy to communicate with, so that a manager could use it as a tool for reasoning rather than receive it as a verdict. Shim et al. (2002) and Arnott and Pervan (2005) traced how the field developed from these beginnings into model-driven systems, executive information systems, data warehouses and group decision support, and later into business intelligence and analytics.
From reporting to delivering insight
Many systems described as decision support are in practice reporting systems. They present data, often in large quantities, and leave the user to find what matters. Eppler and Mengis (2004) reviewed research on information overload and found that when the amount of information exceeds a person's capacity to process it, decision quality falls rather than rises. A dashboard with fifty indicators may contain everything a manager needs and still fail to help, because it does not say which of the fifty deserve attention today.
A system designed around insight delivery reverses the emphasis. It starts from the decision rather than the data. It asks who makes the decision, when, with what options, and what they would need to know to choose better. It then selects, interprets and presents information to answer that need. Few (2006) argued that a well-designed dashboard should show the most important information needed to achieve objectives on a single screen so that it can be monitored at a glance. The same principle applies to more complex systems: what is left out is as important as what is shown.
What the evidence says works
Some of the strongest evidence on decision support comes from medicine, where clinical decision-support systems have been tested in many randomised trials. Garg et al. (2005) reviewed 100 studies and found that such systems improved practitioner performance in most of the studies that measured it, although effects on patient outcomes were less often demonstrated. Kawamoto et al. (2005) analysed 70 trials to identify the features associated with success. Four features stood out: the support was provided automatically as part of the clinician's normal workflow, it was delivered at the time and place the decision was made, it gave a specific recommendation rather than only an assessment, and it was computer-based. Systems that provided support automatically within the workflow were far more likely to improve practice than systems that required the user to look for advice.
These findings translate directly to business settings. An insight that a sales manager must remember to look up in a separate tool is far less likely to be used than one that appears in the system the manager already uses to plan visits. A risk score is less useful than a recommendation that explains what to do about the risk.
Research in marketing supports the same conclusion. Lilien et al. (2004) studied managers using a decision-support system for allocating marketing resources and found that it improved the objective quality of their decisions, but that users did not always perceive the improvement and did not rate the system more highly than those without it. Kayande et al. (2009) showed that systems which gave users feedback on how their own mental model differed from the system's model helped users understand and accept the system's recommendations. The implication is that a system must not only produce good advice but also help users see why it is good.
The human side: acceptance and trust
A decision-support system works only if people use it. Davis (1989) found that perceived usefulness and perceived ease of use predicted whether people adopted new information systems, with usefulness having the stronger influence. Todd and Benbasat (1992) found that decision aids tended to be used in ways that reduced effort rather than improved decision quality: people used the aid to reach a satisfactory decision faster, not necessarily a better one. Designers should therefore make the better option the easier one to follow.
Trust is a second issue. Dietvorst et al. (2015) found that people were less willing to rely on an algorithm after seeing it make a mistake, even when it outperformed human forecasters, a pattern they called algorithm aversion. In a later study, Dietvorst et al. (2018) found that people were much more willing to use an imperfect algorithm if they could adjust its output, even slightly. Logg et al. (2019) found that in some conditions people preferred advice labelled as coming from an algorithm to advice from a person. Together, these findings suggest that systems should present recommendations as advice that the user can examine and adjust, not as instructions to be obeyed or rejected.
The risks of support
Decision support can also create problems. Parasuraman and Manzey (2010) reviewed research on automation complacency and automation bias, the tendency to rely on automated advice without sufficient checking, which can lead people to miss errors the system makes and to act on wrong recommendations. Bainbridge (1983) described the "ironies of automation": the more a system takes over routine tasks, the less practice people have in handling the unusual cases the system cannot, which are precisely the cases where human judgement matters most.
Alert fatigue is a related risk. When a system issues too many warnings, people learn to ignore them. Ancker et al. (2017) studied a clinical decision-support system and found that clinicians were more likely to accept an alert when they had seen fewer alerts recently, and that repeated alerts for the same patient were accepted less often. The design lesson is that each alert spends a limited supply of attention. A system that warns about everything effectively warns about nothing.
An example
Consider a subscription business that wants to reduce cancellations. Its first system was a weekly report listing customers with a high predicted probability of cancelling. Account managers received the report but seldom acted on it, because it arrived separately from their daily work, did not say why each customer was at risk, and listed so many customers that there was no clear place to start.
The redesigned system starts from the decision the account manager makes each morning: whom to contact and what to offer. It places a short list of the five customers most worth contacting directly in the customer-management tool the manager already uses. For each, it shows the main reasons for the risk, such as a fall in usage or an unresolved complaint, and a suggested action, such as a training call or a service credit. The manager can accept, change or dismiss each suggestion and records the result. Those results are fed back to show which actions work for which kinds of customers, and the suggestions improve over time.
Common mistakes
The most common mistake is building the system around the available data rather than the decision. A second is measuring success by the number of users or reports produced rather than by changes in decisions and outcomes. A third is presenting predictions without reasons, which makes it hard for users to judge when to trust them. A fourth is adding warnings and indicators over time without removing any, until the system becomes the kind of overloaded display it was meant to replace. A fifth is treating the system as finished at launch. Because conditions change, the models and rules inside a decision-support system need regular checking against outcomes.
Practical steps
Teams designing a decision-support system can start with a few questions. Which decision is the system meant to improve, who makes it and how often? What information would change that decision, and what is merely interesting? Where in their existing work do decision makers look, and can the insight appear there? Can the system give a specific recommendation and a short reason for it? How will users be able to question, adjust or override the recommendation, and how will those choices be recorded? How many alerts can users reasonably handle, and which ones will be removed when new ones are added? And how will the organisation know, after six months, whether decisions have improved?
Conclusion
Decision-support systems began with the idea that computers could help managers with problems that were partly structured and partly a matter of judgement (Gorry & Scott Morton, 1971). That idea remains sound, but the experience of the past fifty years shows that the analysis is only half the task. The other half is delivery: placing a clear, explained recommendation in the decision maker's workflow at the moment of choice, allowing the person to adjust it, limiting the demands on their attention and learning from the results. Systems designed in this way turn insight into better decisions. Systems that only present data leave that work to their users, who often do not have the time to do it.
References
Ancker, J. S., Edwards, A., Nosal, S., Hauser, D., Mauer, E., & Kaushal, R. (2017). Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Medical Informatics and Decision Making, 17, Article 36. https://doi.org/10.1186/s12911-017-0430-8
Arnott, D., & Pervan, G. (2005). A critical analysis of decision support systems research. Journal of Information Technology, 20(2), 67–87. https://doi.org/10.1057/palgrave.jit.2000035
Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779. https://doi.org/10.1016/0005-1098(83)90046-8
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114–126. https://doi.org/10.1037/xge0000033
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018). Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Management Science, 64(3), 1155–1170. https://doi.org/10.1287/mnsc.2016.2643
Eppler, M. J., & Mengis, J. (2004). The concept of information overload: A review of literature from organization science, accounting, marketing, MIS, and related disciplines. The Information Society, 20(5), 325–344. https://doi.org/10.1080/01972240490507974
Few, S. (2006). Information dashboard design: The effective visual communication of data. O'Reilly.
Garg, A. X., Adhikari, N. K. J., McDonald, H., Rosas-Arellano, M. P., Devereaux, P. J., Beyene, J., Sam, J., & Haynes, R. B. (2005). Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: A systematic review. JAMA, 293(10), 1223–1238. https://doi.org/10.1001/jama.293.10.1223
Gorry, G. A., & Scott Morton, M. S. (1971). A framework for management information systems. Sloan Management Review, 13(1), 55–70.
Kawamoto, K., Houlihan, C. A., Balas, E. A., & Lobach, D. F. (2005). Improving clinical practice using clinical decision support systems: A systematic review of trials to identify features critical to success. BMJ, 330(7494), 765. https://doi.org/10.1136/bmj.38398.500764.8F
Kayande, U., De Bruyn, A., Lilien, G. L., Rangaswamy, A., & van Bruggen, G. H. (2009). How incorporating feedback mechanisms in a DSS affects DSS evaluations. Information Systems Research, 20(4), 527–546. https://doi.org/10.1287/isre.1080.0198
Lilien, G. L., Rangaswamy, A., van Bruggen, G. H., & Starke, K. (2004). DSS effectiveness in marketing resource allocation decisions: Reality vs. perception. Information Systems Research, 15(3), 216–235. https://doi.org/10.1287/isre.1040.0026
Little, J. D. C. (1970). Models and managers: The concept of a decision calculus. Management Science, 16(8), B466–B485. https://doi.org/10.1287/mnsc.16.8.B466
Logg, J. M., Minson, J. A., & Moore, D. A. (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes, 151, 90–103. https://doi.org/10.1016/j.obhdp.2018.12.005
Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. https://doi.org/10.1177/0018720810376055
Shim, J. P., Warkentin, M., Courtney, J. F., Power, D. J., Sharda, R., & Carlsson, C. (2002). Past, present, and future of decision support technology. Decision Support Systems, 33(2), 111–126. https://doi.org/10.1016/S0167-9236(01)00139-7
Simon, H. A. (1960). The new science of management decision. Harper & Row.
Todd, P., & Benbasat, I. (1992). The use of information in decision making: An experimental investigation of the impact of computer-based decision aids. MIS Quarterly, 16(3), 373–393. https://doi.org/10.2307/249534