Insights · Communicating insight
From Insight to Innovation: Linking Understanding to New Products
How customer insight leads to new products and services, covering the evidence on what drives new product success, jobs and outcomes, lead users, empathic observation, the Kano model, and how to test insight-based concepts before committing to them.
New products fail often. Estimates vary, and some widely quoted figures are exaggerated: Castellion and Markham (2013) reviewed the evidence and concluded that the commonly cited claims of failure rates of 80% or more were unsupported, and that the best estimates were closer to 40%. Even so, a large share of the effort that goes into new products is wasted. One of the most consistent findings in research on innovation is that products built on a deep understanding of customers succeed more often than those that are not. This article explains how insight leads to innovation and how organisations can make the link more reliable.
Insight as a driver of new product success
Cooper and Kleinschmidt (1987) studied new industrial products and found that product superiority, meaning a product that offers customers unique benefits, and a strong understanding of the market before development began were among the factors most closely associated with success. Henard and Szymanski (2001) combined the results of many studies of new product performance in a meta-analysis and found that the product's advantage to customers, its fit with customer needs and the quality of market research and other pre-development activities were among the strongest predictors of success.
These findings point to a sequence: understanding customers well enough to identify a need that is unmet or poorly met, then designing a product that meets it clearly better than alternatives.
Responding to needs and anticipating them
Customers can describe some of their needs, but not all. Narver et al. (2004) distinguished a "responsive" market orientation, which seeks to understand and meet customers' expressed needs, from a "proactive" one, which seeks to understand and meet needs customers have not articulated. They found that proactive market orientation was positively related to new product success, while responsive orientation on its own was not, because responding to expressed needs is what competitors do too.
Christensen (1997) showed how listening closely to existing customers can lead established firms to miss innovations that initially appeal to other customers. Insight for innovation therefore requires more than asking customers what they want.
Methods that link insight to innovation
Jobs and outcomes
Christensen et al. (2016) argued that customers "hire" products to get a "job" done, a type of progress they are trying to make in a particular situation, and that innovation succeeds when it helps people make that progress better. Ulwick (2002) proposed that customers should be asked not for solutions but for the outcomes they want when doing a job, which can then be rated for importance and current satisfaction to reveal underserved outcomes. The job, not the product category, defines the competition and the opportunity.
Lead users
Some customers face needs earlier and more intensely than the market as a whole. von Hippel (1986) called them "lead users" and proposed that they are a valuable source of new product ideas, because they experience needs that will later become general and often develop their own solutions. Lilien et al. (2002) compared lead user projects with conventional idea generation at 3M and found that the lead user projects produced ideas with much higher forecast sales.
Empathic observation
Leonard and Rayport (1997) described "empathic design", in which researchers observe customers using products in their own environments to identify needs customers would not think to mention, such as workarounds, unnoticed frustrations and unexpected uses. Observation often reveals the gap between what people say they do and what they actually do. It is particularly useful for products used in the home or at work, where routines are so familiar that people no longer notice the effort they involve. A researcher who watches someone assemble a meal, set up a device or complete a form can see hesitations, mistakes and improvised solutions that the person would never describe in an interview, because to them these are simply part of the task. Leonard and Rayport recommended that observation be carried out by small teams with different backgrounds, who can notice different things, and that observations be captured visually, through photographs or video, so that they can be shared with designers and engineers.
Voice of the customer
Griffin and Hauser (1993) studied methods for capturing customer needs for product development and found that one-to-one interviews with 20 to 30 customers identified most of the needs that larger samples would find. They also described how needs can be organised into a hierarchy and linked to design decisions. Hauser and Clausing (1988) described the "house of quality", a matrix that connects customer needs with engineering characteristics so that design decisions can be traced to what customers value.
Customers as idea sources
Customers can also generate ideas directly. Poetz and Schreier (2012) compared product ideas from users with ideas from a company's professionals and found that users' ideas were rated higher on novelty and customer benefit, though lower on feasibility. Bayus (2013) studied an online community in which customers suggested product ideas to a computer company and found that customers who had already had an idea implemented were unlikely to repeat their success, apparently because their later ideas tended to resemble the ideas that had already been adopted.
Not all needs are equal
The Kano model distinguishes types of product attributes by how they affect satisfaction (Kano et al., 1984). "Must-be" attributes cause dissatisfaction if absent but add little satisfaction when present: customers expect them. "Performance" attributes increase satisfaction in proportion to how well they are delivered. "Attractive" attributes delight customers when present but cause no dissatisfaction when absent, often because customers did not expect them.
For innovation, attractive attributes are especially important, because they distinguish a product from competitors. But they are also the hardest to discover by asking, because customers do not expect them. Over time, attractive attributes tend to become performance and then must-be attributes, as competitors copy them and customers come to expect them.
From insight to concept
An insight becomes an innovation through a series of steps, each of which can be tested.
Insight. State the need clearly, with the evidence for it: who has it, in what situation, how they currently meet it and why that is unsatisfactory.
Opportunity. Estimate how many people have the need, how much it matters to them and what they currently spend to meet it.
Concept. Describe a proposed solution from the customer's point of view: what it does for them, not how it works.
Test. Test the concept with real customers, starting with cheap methods. Thomke (2003) argued that the ability to experiment quickly and cheaply is central to innovation, and that organisations should test early and often. Ries (2011) proposed building a "minimum viable product", the simplest version that allows the team to learn whether customers want the product, and measuring how real customers respond. Brown (2008) described how design thinking uses quick prototypes to learn from customers before committing to a design.
Launch and learn. Measure how customers actually use the product, compare the results with the original insight, and refine.
Cooper (1990) described the "stage-gate" process, in which new product projects pass through a series of stages separated by gates at which managers decide, on the evidence available, whether to continue. Insight provides the evidence at each gate, and the gates provide points at which ideas that do not hold up can be stopped before large sums are spent.
An example
Consider a hypothetical maker of garden tools that wanted to grow beyond its core range. Satisfaction surveys showed customers were happy with existing products, and suggestions for improvement were minor: stronger handles, more colours. These were responsive insights, and competitors were already acting on them.
The company then ran a different kind of research. Researchers visited 25 households and watched people garden, and they interviewed a group of lead users: allotment holders and community gardeners who grew food intensively in small spaces. Observation showed that many people struggled with jobs that involved kneeling and bending, and had improvised solutions such as folded towels, upturned buckets and tools with taped-on extensions. The lead users had gone further, building raised beds and modifying tools to work standing up. None of the survey respondents had mentioned these problems, because they regarded them as part of gardening rather than as something a product could solve.
The insight was stated as a job: "help me keep gardening comfortably as I get older, without giving up the tasks I enjoy." A quick survey showed that the need was common among older gardeners, who were also the company's highest-spending customers. The team developed three concepts, tested rough prototypes in a dozen gardens, dropped one that users found awkward and refined the other two. The resulting range of long-handled and kneeling-support tools became one of the company's fastest-growing lines, and its success was traced, at each stage-gate review, to the original observations.
Keeping a portfolio of needs
Organisations that innovate consistently often keep a shared record of unmet needs, with the evidence for each, its estimated size and its status. New research adds to the record; product teams draw on it when planning; and each need is linked to any concepts developed to meet it. This makes insight a continuing source of innovation rather than a one-off input, and helps prevent valuable findings from being lost when a project ends or the people involved move on.
Why the link breaks
The link between insight and innovation often fails for organisational reasons rather than research ones. Insight teams and product teams may work separately, so that research is commissioned after products have been designed, to validate rather than inform them. Findings may be presented in a form product teams cannot use, such as satisfaction scores rather than specific unmet needs. And organisations may favour incremental improvements to existing products, which are easier to justify, over new ideas whose market is uncertain.
Practices that help include involving product teams in research, expressing insights as specific needs in specific situations, maintaining a shared record of unmet needs that product teams can draw on, and reviewing new product decisions against the insight that prompted them.
Key takeaways
Products built on a deep understanding of customers succeed more often, particularly when they offer clear advantages and meet needs customers may not have expressed. Methods that link insight to innovation include analysing the jobs customers are trying to do and the outcomes they want, working with lead users, observing customers empathically, capturing the voice of the customer and inviting customer ideas. The Kano model shows that not all needs are equal, and that attractive attributes customers do not expect are especially valuable but hard to discover by asking. Insight becomes innovation through a series of testable steps from need to opportunity, concept, test and launch. The link often breaks for organisational reasons, which can be addressed by involving product teams in research and expressing insights as specific unmet needs.
References
Bayus, B. L. (2013). Crowdsourcing new product ideas over time: An analysis of the Dell IdeaStorm community. Management Science, 59(1), 226–244. https://doi.org/10.1287/mnsc.1120.0599
Brown, T. (2008). Design thinking. Harvard Business Review, 86(6), 84–92.
Castellion, G., & Markham, S. K. (2013). Perspective: New product failure rates: Influence of argumentum ad populum and self-interest. Journal of Product Innovation Management, 30(5), 976–979. https://doi.org/10.1111/j.1540-5885.2012.01009.x
Christensen, C. M. (1997). The innovator's dilemma: When new technologies cause great firms to fail. Harvard Business School Press.
Christensen, C. M., Hall, T., Dillon, K., & Duncan, D. S. (2016). Competing against luck: The story of innovation and customer choice. HarperBusiness.
Cooper, R. G. (1990). Stage-gate systems: A new tool for managing new products. Business Horizons, 33(3), 44–54. https://doi.org/10.1016/0007-6813(90)90040-I
Cooper, R. G., & Kleinschmidt, E. J. (1987). New products: What separates winners from losers? Journal of Product Innovation Management, 4(3), 169–184. https://doi.org/10.1111/1540-5885.430169
Griffin, A., & Hauser, J. R. (1993). The voice of the customer. Marketing Science, 12(1), 1–27. https://doi.org/10.1287/mksc.12.1.1
Hauser, J. R., & Clausing, D. (1988). The house of quality. Harvard Business Review, 66(3), 63–73.
Henard, D. H., & Szymanski, D. M. (2001). Why some new products are more successful than others. Journal of Marketing Research, 38(3), 362–375. https://doi.org/10.1509/jmkr.38.3.362.18861
Kano, N., Seraku, N., Takahashi, F., & Tsuji, S. (1984). Attractive quality and must-be quality. Journal of the Japanese Society for Quality Control, 14(2), 39–48.
Leonard, D., & Rayport, J. F. (1997). Spark innovation through empathic design. Harvard Business Review, 75(6), 102–113.
Lilien, G. L., Morrison, P. D., Searls, K., Sonnack, M., & von Hippel, E. (2002). Performance assessment of the lead user idea-generation process for new product development. Management Science, 48(8), 1042–1059. https://doi.org/10.1287/mnsc.48.8.1042.171
Narver, J. C., Slater, S. F., & MacLachlan, D. L. (2004). Responsive and proactive market orientation and new-product success. Journal of Product Innovation Management, 21(5), 334–347. https://doi.org/10.1111/j.0737-6782.2004.00086.x
Poetz, M. K., & Schreier, M. (2012). The value of crowdsourcing: Can users really compete with professionals in generating new product ideas? Journal of Product Innovation Management, 29(2), 245–256. https://doi.org/10.1111/j.1540-5885.2011.00893.x
Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation to create radically successful businesses. Crown Business.
Thomke, S. H. (2003). Experimentation matters: Unlocking the potential of new technologies for innovation. Harvard Business School Press.
Ulwick, A. W. (2002). Turn customer input into innovation. Harvard Business Review, 80(1), 91–97.
von Hippel, E. (1986). Lead users: A source of novel product concepts. Management Science, 32(7), 791–805. https://doi.org/10.1287/mnsc.32.7.791