Skip to content

Insights · The insight function

The Rise of the Insights Function in Modern Organisations

How the in-house market research department evolved into today's insights function, what drove the change, what research says about its value, and what distinguishes an insights function that influences decisions from one that only supplies data.

ModQu research article · 1,767 words · 20 references (APA 7th edition)

Download PDF

A century ago, very few companies had anyone whose job was to study customers systematically. Today most large consumer businesses, and many business-to-business firms, have a dedicated function for it, usually called consumer insight, customer insight or simply insights. Its leaders may sit near the top of the organisation, its budgets can run to millions, and it draws on surveys, behavioural data, social media, experiments and analytics. This article traces how the function developed, explains the forces that shaped it, and summarises what research says about the conditions under which it adds value.

From commercial research to market research departments

Organised commercial research began in the early twentieth century. In 1911 the Curtis Publishing Company in the United States set up a commercial research division under Charles Coolidge Parlin, whose studies of markets for products advertised in Curtis magazines are often described as the start of market research as a business activity (Lockley, 1950). Over the following decades, advertising agencies, manufacturers and specialist research firms developed consumer surveys, retail audits and panels. By the middle of the century, many large consumer-goods companies had their own market research departments.

These departments were primarily technical. They designed studies, commissioned fieldwork from agencies, analysed results and reported findings to marketing managers, who then made the decisions. Researchers were valued for methodological expertise and independence. Their influence depended on whether managers chose to use their reports, and studies of research use found that this was far from guaranteed. Deshpandé and Zaltman (1982) showed that managers' use of market research depended heavily on the perceived quality of the research and on the relationship between researchers and managers, not only on the findings.

The market orientation movement

A shift in thinking came in the 1990s. Researchers in marketing proposed that successful companies are market oriented: they generate intelligence about customers and competitors, spread it across departments, and respond to it in a coordinated way (Kohli & Jaworski, 1990; Narver & Slater, 1990). Market intelligence, in this view, is not the property of one department. It is an organisation-wide capability.

Empirical research supported the idea. Narver and Slater (1990) found a positive relationship between market orientation and profitability, and Jaworski and Kohli (1993) linked it to overall business performance. A later meta-analysis by Kirca, Jayachandran and Bearden (2005), combining results from many studies, confirmed a positive association between market orientation and performance across industries and countries. Day (1994) described "market sensing", the ability to learn about markets ahead of competitors, as one of the distinctive capabilities of market-driven firms.

The market orientation literature also linked market intelligence to organisational learning. Sinkula (1994) described how organisations acquire, distribute and interpret market information, and how they store what they learn in organisational memory. Slater and Narver (1995) argued that market orientation is most valuable when combined with a culture of learning that questions existing assumptions. Moorman (1995) found that the way organisations process market information was related to the success of their new products.

These ideas changed the job description of the research department. Producing accurate studies was no longer enough. The department was expected to help the whole organisation understand customers and act on that understanding.

From research to insight

From the late 1990s onwards, many companies renamed their market research departments as consumer or customer insight teams. The change was more than cosmetic. Said, Macdonald, Wilson and Marcos (2015) studied how large organisations generate and use customer insight and found that managers distinguished sharply between information, which was plentiful, and insight, which was new understanding that was shared and acted upon. The insight function was expected to integrate many sources of information, interpret them in the light of business problems and influence decisions directly.

Four stages in the evolution of the function: commercial research (from 1911, studies of markets), market research departments (mid-twentieth century, technical studies for marketing), market intelligence and orientation (1990s, intelligence shared across the firm), and insights and analytics (2000s onward, integrated sources, influence on strategy).
Figure 1. Four broad stages in the development of the insights function. Each stage added responsibilities rather than replacing earlier ones: technical research quality still matters. Diagram by ModQu, based on Lockley (1950), Kohli and Jaworski (1990) and Said et al. (2015).

Several pressures drove the change.

More data. Digital channels, loyalty programmes and transaction systems produced far more customer data than surveys alone. Wedel and Kannan (2016) describe how marketing analytics evolved to handle these data-rich environments, combining customer records, digital behaviour and research. Someone had to make sense of the combined picture.

Demand for accountability. Senior executives increasingly expected marketing and research spending to show measurable returns. Moorman and Rust (1999) found that the marketing function contributed to company performance beyond the effect of market orientation when it connected customers to products, service delivery and financial accountability. Verhoef and Leeflang (2009) found that the influence of marketing departments within firms was associated with their accountability and innovativeness, and that greater influence was linked to stronger market orientation and better performance.

Competition on analytics. Davenport (2006) described companies that compete on analytics, using data and quantitative analysis as a central part of strategy. Brynjolfsson, Hitt and Kim (2011) found that firms that reported basing decisions on data had higher productivity than would be expected from their other investments. Germann, Lilien and Rangaswamy (2013) surveyed senior managers in large firms and found that greater deployment of marketing analytics was associated with better firm performance, particularly in industries with intense competition and fast-changing customer preferences.

A changing role for marketing. Webster (1992) argued that marketing was moving from a functional department to a way of managing relationships with customers across the whole organisation. The insights function followed the same path: from a service department to a contributor to strategy.

What a modern insights function does

Modern insights functions vary widely, but most combine several roles.

The balance between these roles differs by company. Some insights functions remain close to their research origins; others have merged with data science and analytics teams.

Where the function sits

Companies place their insights function in different parts of the organisation, and the choice affects what it can do. When the function sits within marketing, it is usually close to brand and communication decisions but may have less reach into product development, operations or customer service. When it sits within a strategy or planning function, it may have more influence on long-term choices but less contact with day-to-day decisions. Some companies have created a central insights or analytics unit serving the whole business; others distribute insight specialists across business units with a small central team setting standards.

There is no single right answer, but research on market orientation suggests a principle. Because market intelligence is valuable only when it is shared and acted on across functions (Kohli & Jaworski, 1990), the insights function should be placed and connected so that its work reaches all the departments whose decisions affect customers. A function that reports only to marketing risks producing insight that is used only by marketing.

An illustration of the shift

The change from research department to insights function can be seen in how a typical request is handled. In the older model, a brand manager might ask the research department for a survey of awareness and attitudes before a campaign. The department would design the questionnaire, commission an agency, analyse the results and deliver a report, which the brand manager would read and interpret.

In the newer model, the insights team would start by asking what decision the campaign was meant to support and what the business already knew. It might combine the survey with sales data, search trends and customer service records, test competing explanations for weak performance and present a small number of insights with recommended actions to a group that included marketing, sales and product managers. The survey might be the same; the role around it is different.

When the insights function adds value

Research on information use suggests that the value of an insights function depends less on the volume of research it produces than on whether its work is used. Menon and Varadarajan (1992) distinguished between research used directly in decisions and research that changes how managers think about problems; both matter, but both depend on a relationship of trust between researchers and decision-makers. Weiss (1979) observed that research often influences decisions gradually, by reshaping the way problems are understood, rather than by settling individual questions.

Several conditions recur in the literature:

  1. Proximity to decisions. Insights teams that understand the business problems and are involved early in decision processes produce work that is used more (Deshpandé & Zaltman, 1982).
  2. Integration across sources. Organisations that combine many sources of customer information, rather than keeping them in separate departments, are better placed to generate insight (Said et al., 2015).
  3. A learning culture. Market intelligence is most valuable in organisations willing to question their assumptions (Slater & Narver, 1995).
  4. Accountability. Functions that can show how their work affected decisions and results gain influence (Verhoef & Leeflang, 2009).
  5. Methodological quality. The move from research to insight does not reduce the importance of sound methods. Insights built on poor data are more persuasive than raw findings, and therefore more dangerous.

Tensions and challenges

The rise of the insights function has brought new tensions. Closer involvement in decisions can threaten the independence that made research credible: researchers who see themselves as advocates may be tempted to present findings selectively. The expansion of data sources creates overlaps with data science, digital analytics and customer experience teams, raising questions about who owns the customer view. And the expectation of influence can lead to an emphasis on compelling stories over careful evidence.

Successful functions manage these tensions by combining analytical rigour with business engagement: close enough to decisions to be relevant, independent enough to be trusted.

Key takeaways

The insights function grew from early commercial research, through mid-century market research departments, into an organisation-wide capability for understanding customers. The market orientation research of the 1990s, the explosion of customer data, the demand for accountability and the rise of analytics all pushed it from producing reports toward shaping decisions. Research suggests that the function adds most value when it is close to decisions, integrates many sources, works within a learning culture, can demonstrate its impact and maintains rigorous methods.

References

Brynjolfsson, E., Hitt, L. M., & Kim, H. H. (2011). Strength in numbers: How does data-driven decisionmaking affect firm performance? (SSRN Working Paper). https://doi.org/10.2139/ssrn.1819486

Davenport, T. H. (2006). Competing on analytics. Harvard Business Review, 84(1), 98–107.

Day, G. S. (1994). The capabilities of market-driven organizations. Journal of Marketing, 58(4), 37–52. https://doi.org/10.1177/002224299405800404

Deshpandé, R., & Zaltman, G. (1982). Factors affecting the use of market research information: A path analysis. Journal of Marketing Research, 19(1), 14–31. https://doi.org/10.1177/002224378201900102

Germann, F., Lilien, G. L., & Rangaswamy, A. (2013). Performance implications of deploying marketing analytics. International Journal of Research in Marketing, 30(2), 114–128. https://doi.org/10.1016/j.ijresmar.2012.10.001

Jaworski, B. J., & Kohli, A. K. (1993). Market orientation: Antecedents and consequences. Journal of Marketing, 57(3), 53–70. https://doi.org/10.1177/002224299305700304

Kirca, A. H., Jayachandran, S., & Bearden, W. O. (2005). Market orientation: A meta-analytic review and assessment of its antecedents and impact on performance. Journal of Marketing, 69(2), 24–41. https://doi.org/10.1509/jmkg.69.2.24.60761

Kohli, A. K., & Jaworski, B. J. (1990). Market orientation: The construct, research propositions, and managerial implications. Journal of Marketing, 54(2), 1–18. https://doi.org/10.1177/002224299005400201

Lockley, L. C. (1950). Notes on the history of marketing research. Journal of Marketing, 14(5), 733–736. https://doi.org/10.1177/002224295001400508

Menon, A., & Varadarajan, P. R. (1992). A model of marketing knowledge use within firms. Journal of Marketing, 56(4), 53–71. https://doi.org/10.1177/002224299205600405

Moorman, C. (1995). Organizational market information processes: Cultural antecedents and new product outcomes. Journal of Marketing Research, 32(3), 318–335. https://doi.org/10.1177/002224379503200307

Moorman, C., & Rust, R. T. (1999). The role of marketing. Journal of Marketing, 63(Special Issue), 180–197. https://doi.org/10.1177/00222429990634s117

Narver, J. C., & Slater, S. F. (1990). The effect of a market orientation on business profitability. Journal of Marketing, 54(4), 20–35. https://doi.org/10.1177/002224299005400403

Said, E., Macdonald, E. K., Wilson, H. N., & Marcos, J. (2015). How organisations generate and use customer insight. Journal of Marketing Management, 31(9–10), 1158–1179. https://doi.org/10.1080/0267257X.2015.1037785

Sinkula, J. M. (1994). Market information processing and organizational learning. Journal of Marketing, 58(1), 35–45. https://doi.org/10.1177/002224299405800103

Slater, S. F., & Narver, J. C. (1995). Market orientation and the learning organization. Journal of Marketing, 59(3), 63–74. https://doi.org/10.1177/002224299505900306

Verhoef, P. C., & Leeflang, P. S. H. (2009). Understanding the marketing department's influence within the firm. Journal of Marketing, 73(2), 14–37. https://doi.org/10.1509/jmkg.73.2.14

Webster, F. E., Jr. (1992). The changing role of marketing in the corporation. Journal of Marketing, 56(4), 1–17. https://doi.org/10.1177/002224299205600402

Wedel, M., & Kannan, P. K. (2016). Marketing analytics for data-rich environments. Journal of Marketing, 80(6), 97–121. https://doi.org/10.1509/jm.15.0413

Weiss, C. H. (1979). The many meanings of research utilization. Public Administration Review, 39(5), 426–431. https://doi.org/10.2307/3109916