Skip to content

Insights · The insight function

From Market Research Department to Insights Team

What actually changes when a market research department becomes an insights team: its purpose, skills, data, outputs, relationships and measures of success. This article draws on research about market orientation, organisational learning and knowledge transfer to explain the transition and the common reasons it fails.

ModQu research article · 2,826 words · 24 references (APA 7th edition)

Download PDF

Over the past three decades, many organisations have renamed their market research departments as insights teams. In some, the new name reflected a real change in role, skills and influence. In others, the same people continued to produce the same reports under a new label. The difference matters. A research department that merely changes its name does not gain influence, while one that genuinely becomes an insights team can change how an organisation understands its customers and makes decisions.

This article examines what the transition involves. It describes the traditional research department, the forces that pushed it to change, the specific shifts that distinguish an insights team, and the organisational conditions that research suggests are needed for the change to succeed.

The traditional market research department

Formal market research began in the early twentieth century, when companies such as the Curtis Publishing Company created divisions to study markets systematically (Lockley, 1950). By the middle of the century, large consumer-goods firms had in-house departments staffed by specialists in survey design, sampling and statistics. Their typical role had four features.

A service function. The department responded to requests from marketing managers. Managers decided what questions to ask and what to do with the answers.

Project-based work. Research was organised as discrete studies: a concept test, an advertising pre-test, a segmentation, a satisfaction tracker. Each produced a report.

Methodological authority. Researchers were valued for technical expertise and independence. Their credibility rested on sound sampling, careful questionnaires and correct statistics.

Distance from decisions. Researchers were often not present when decisions were made. Their influence depended on whether managers read, trusted and used their reports. Deshpandé and Zaltman (1982) found that research use depended strongly on managers' perceptions of research quality and on the interaction between researchers and managers.

This model produced a great deal of valuable research. But it had a structural weakness: the people who understood the data best were often furthest from the decisions the data were meant to inform.

Forces for change

Several developments from the 1990s onward pushed research departments to change.

Market orientation

Research on market orientation showed that companies perform better when market intelligence is generated, spread across the organisation and acted on in a coordinated way (Kohli & Jaworski, 1990; Narver & Slater, 1990). Deshpandé, Farley and Webster (1993) found that in Japanese firms, customer orientation as judged by customers, and an organisational culture that valued market responsiveness, were associated with better performance. In this view, understanding customers was a responsibility of the whole organisation, and the research department was one contributor among many. Day (1994) called the ability to sense market changes ahead of competitors a distinctive capability, rooted in organisational processes rather than in individual studies.

Organisational learning

Research on organisational learning offered a further frame. Huber (1991) described organisational learning as four connected processes: acquiring knowledge, distributing information, interpreting it and storing it in organisational memory. Sinkula (1994) applied this framework to market information. A traditional research department focused mainly on acquisition. Distribution, interpretation and memory were left to others, or to chance. Slater and Narver (1995) argued that market orientation creates value mainly when combined with a learning culture that questions existing assumptions.

Data abundance

Customer data multiplied. Transaction records, loyalty programmes, websites, mobile apps, call centres and social media all generated information about customers outside the research department. Wedel and Kannan (2016) describe how marketing analytics developed to handle these data-rich environments. Moorman (1995) had already shown that the way organisations acquire, transmit and use market information matters for outcomes such as new product success; with many more sources, these processes became harder and more important. A department that controlled only surveys controlled a shrinking share of what the organisation knew about its customers.

Accountability

Senior leaders increasingly expected every function to show its contribution to results. Verhoef and Leeflang (2009) found that the influence of marketing departments within firms depended partly on their accountability, their ability to link activities to financial outcomes. Research departments faced the same pressure: they needed to show that their work changed decisions.

What changes in the transition

The transition from research department to insights team involves at least six shifts. Each can happen independently, which is why some renamed teams change a great deal and others hardly at all.

Comparison table of the traditional research department and the insights team across six dimensions: purpose (answer questions asked versus improve decisions), main output (reports versus insights and recommendations), data (mainly own surveys versus many integrated sources), skills (research methods versus methods plus business, analytics and storytelling), relationships (service provider versus partner in decisions) and success measure (quality and timeliness versus impact on decisions).
Figure 1. Six shifts that distinguish an insights team from a traditional research department. Changing the name without these shifts changes little. Diagram by ModQu, based on Said et al. (2015) and the market orientation literature.

1. From answering questions to improving decisions

The research department answered the questions it was asked. The insights team starts from the decision. It asks what choice the business faces, what it already knows and what it would need to learn to choose better. This sometimes means questioning the original brief, because the question a manager asks is not always the one that matters most.

2. From reports to insights and recommendations

The research department's output was the report, often long and comprehensive. The insights team's output is a smaller number of insights, each explaining why customers behave as they do and what that implies, with a recommendation. Said, Macdonald, Wilson and Marcos (2015) found that organisations distinguished clearly between information and insight, and valued insight that was new, shared and acted on.

3. From owned surveys to integrated sources

The research department worked mainly with data it collected. The insights team integrates surveys with transaction data, digital analytics, customer service records, social media and external information. This requires new relationships with analytics, IT and other teams that hold customer data, and new skills in combining sources whose strengths and biases differ.

4. From research methods to a broader skill set

Methodological expertise remains essential, but it is no longer enough. Insights teams also need business understanding, analytical skills that span statistics and data science, the ability to synthesise evidence from many sources, and the ability to communicate persuasively. Germann, Lilien and Rangaswamy (2013) found that the performance benefits of marketing analytics depended on the deployment of analytical capability across the firm, which in turn depended on people, skills and top management support.

5. From service provider to partner

The research department served internal clients. The insights team works alongside them, ideally from the start of a project to the decision and beyond. Menon and Varadarajan (1992) emphasised that how research is used depends on the relationship between researchers and users and on whether research is seen as a source of new understanding rather than only as an answer to a specific question.

6. From quality metrics to impact

The research department measured itself by the quality and timeliness of studies. The insights team also measures its impact: which decisions changed, what resulted, and what the organisation learned. This shift is difficult, because the effect of an insight on a business outcome is rarely easy to isolate, but it is central to the team's credibility.

The role of external agencies

Most organisations continue to buy research from external agencies. The transition changes this relationship too. A traditional department often managed agencies as suppliers of fieldwork and analysis. An insights team typically keeps the work closest to decisions, such as framing problems, synthesising across sources and making recommendations, in-house, and uses agencies for specialised methods, large-scale data collection and independent perspectives. Some organisations build long-term partnerships with a small number of agencies that understand their business well; others use many agencies for specific tasks. Either way, the insights team must be able to judge the quality of external work, because it remains accountable for the insights built on it.

Knowledge management and organisational memory

A recurring weakness of traditional research was that knowledge disappeared after each project. Reports were filed and forgotten, and new studies repeated old questions. Huber (1991) identified organisational memory as one of the core processes of learning, and an insights team must manage it deliberately.

Nonaka (1994) distinguished between explicit knowledge, which can be written down, and tacit knowledge, which resides in people's experience and is hard to articulate. Much of what researchers know about customers is tacit: a feel for how customers talk, which findings are robust, which segments behave unpredictably. Nonaka described how organisations create knowledge by converting between tacit and explicit forms, through shared experience, articulation, combination of documents and learning by doing. Insights teams that rely only on written reports capture only part of what they know.

Transferring knowledge within organisations is harder than it looks. Szulanski (1996) studied the transfer of best practices between units of the same company and found that the main barriers were not motivation but the recipient's lack of capacity to absorb the knowledge, ambiguity about why the practice worked, and difficult relationships between source and recipient. Argote and Ingram (2000) argued that knowledge is embedded in people, tools and tasks, and that moving people is often the most effective way to move knowledge. For insights teams, this suggests that placing insight specialists within business teams, involving business managers in research and building shared tools are more effective than circulating reports. Cohen and Levinthal (1990) made a related point: an organisation's ability to recognise and use new knowledge depends on its existing related knowledge, so investment in customer understanding across the business increases the value of each new insight.

Culture: the hidden requirement

The transition depends on organisational culture as much as on the insights team itself. Several studies point to the same conditions.

Willingness to learn. Slater and Narver (1995) argued that market intelligence creates value when the organisation is willing to question its assumptions. In organisations that use research mainly to confirm decisions already made, an insights team will struggle however skilled it is.

Psychological safety. Edmondson (1999) found that teams whose members felt safe to raise problems and admit mistakes engaged in more learning behaviour and performed better. Insights that challenge existing strategies require an environment in which raising them is not risky.

Balance between exploitation and exploration. March (1991) showed that organisations tend to favour refining what they already know over exploring new possibilities. An insights team that only tracks and optimises existing activities will produce useful but limited insight. Protecting some capacity for exploratory work helps the organisation notice changes before competitors do.

Shaping markets as well as responding to them. Jaworski, Kohli and Sahay (2000) distinguished between being market-driven, responding to customers' existing preferences, and driving markets, changing customers' preferences or the structure of the market. Insights teams can contribute to both, but driving markets requires insight into needs that customers cannot yet articulate, which calls for methods beyond conventional surveys.

A practical roadmap

Organisations that have made the transition successfully tend to move through a recognisable sequence, although the stages overlap.

Stage 1: Clarify the mandate. Senior leaders agree what the insights team is for, which decisions it should inform and how it will work with marketing, product, sales, service and analytics. Without this agreement, the team's requests for access to decisions and data are easily refused.

Stage 2: Map the customer knowledge the organisation already has. Most organisations hold far more customer information than any one team realises: old studies, tracking data, complaint logs, sales notes, web analytics. Mapping these sources reveals duplication, gaps and opportunities for integration. It also shows other teams that the insights function intends to work with their data rather than replace it.

Stage 3: Change the output. The team replaces long reports with short insight statements, each linked to a decision and a recommendation, supported by an evidence appendix for those who want it. This is often the most visible change and the one that most quickly alters how the team is perceived.

Stage 4: Build skills and roles. The team adds analytical, business and communication skills, through training, hiring and partnerships. Research on big data analytics capability suggests that technology is only one part of the capability; the people who use it and the organisational routines around it matter as much (Gupta & George, 2016).

Stage 5: Embed in decision processes. Insights specialists join product teams, planning cycles and leadership meetings. Research becomes a standing input to decisions rather than an occasional commission.

Stage 6: Close the loop. The team records which insights led to which decisions and tracks results. Over time this creates evidence of value and a body of organisational memory that makes each new project faster and better informed.

An illustration

A mid-sized retailer had a research department of four people that ran an annual customer satisfaction survey, a quarterly brand tracker and occasional advertising tests. Its reports were well regarded but rarely discussed by senior managers. When the company appointed a head of customer insight, the first step was not a new study but a series of conversations with directors about the decisions they faced over the coming year. Three stood out: whether to expand home delivery, how to respond to a discount competitor and why repeat visits had fallen among younger customers.

The team then combined its existing survey data with loyalty card records and website analytics to address these questions, presenting each answer on a single page with a recommendation. The analysis of repeat visits, for example, showed that younger customers were not leaving for the discount competitor, as directors had assumed, but were buying online from other retailers because the company's own website did not show stock availability. The finding changed the priority from price matching to website investment. Within two years the insights team reported directly to the leadership group, and its annual survey had been redesigned around the decisions it informed. The illustration is simplified, but it shows that the decisive changes were in mandate, integration and output, not in the research methods themselves.

How to tell whether the transition has happened

Because the label "insights team" is easy to adopt, it helps to have practical signs that the change is real. The following questions can be asked of any team, and the answers are usually clear.

These signs are not a formal scale, but together they show whether the team is producing information for others to interpret or insight that shapes what the organisation does.

Why the transition often fails

Experience and research suggest several common reasons why the transition stalls.

The last point deserves emphasis. The value of an insights team rests on trust. Weiss (1979) showed that research shapes organisations largely by gradually changing how problems are understood. That influence depends on a record of being right. An insights team that sacrifices rigour for influence may gain attention briefly but will lose it when its insights fail.

Key takeaways

The move from market research department to insights team is more than a change of name. It involves shifts in purpose, output, data, skills, relationships and measures of success, from answering questions to improving decisions. Research on market orientation and organisational learning explains why the shift creates value: market intelligence matters most when it is shared, interpreted and acted on across the organisation. Research on knowledge transfer and culture explains why the shift is hard: knowledge is sticky, organisations favour exploitation, and challenging insights need psychological safety. Successful insights teams combine business partnership with undiminished methodological rigour.

References

Argote, L., & Ingram, P. (2000). Knowledge transfer: A basis for competitive advantage in firms. Organizational Behavior and Human Decision Processes, 82(1), 150–169. https://doi.org/10.1006/obhd.2000.2893

Cohen, W. M., & Levinthal, D. A. (1990). Absorptive capacity: A new perspective on learning and innovation. Administrative Science Quarterly, 35(1), 128–152. https://doi.org/10.2307/2393553

Day, G. S. (1994). The capabilities of market-driven organizations. Journal of Marketing, 58(4), 37–52. https://doi.org/10.1177/002224299405800404

Deshpandé, R., & Zaltman, G. (1982). Factors affecting the use of market research information: A path analysis. Journal of Marketing Research, 19(1), 14–31. https://doi.org/10.1177/002224378201900102

Deshpandé, R., Farley, J. U., & Webster, F. E., Jr. (1993). Corporate culture, customer orientation, and innovativeness in Japanese firms: A quadrad analysis. Journal of Marketing, 57(1), 23–37. https://doi.org/10.1177/002224299305700102

Edmondson, A. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44(2), 350–383. https://doi.org/10.2307/2666999

Germann, F., Lilien, G. L., & Rangaswamy, A. (2013). Performance implications of deploying marketing analytics. International Journal of Research in Marketing, 30(2), 114–128. https://doi.org/10.1016/j.ijresmar.2012.10.001

Gupta, M., & George, J. F. (2016). Toward the development of a big data analytics capability. Information & Management, 53(8), 1049–1064. https://doi.org/10.1016/j.im.2016.07.004

Huber, G. P. (1991). Organizational learning: The contributing processes and the literatures. Organization Science, 2(1), 88–115. https://doi.org/10.1287/orsc.2.1.88

Jaworski, B., Kohli, A. K., & Sahay, A. (2000). Market-driven versus driving markets. Journal of the Academy of Marketing Science, 28(1), 45–54. https://doi.org/10.1177/0092070300281005

Kohli, A. K., & Jaworski, B. J. (1990). Market orientation: The construct, research propositions, and managerial implications. Journal of Marketing, 54(2), 1–18. https://doi.org/10.1177/002224299005400201

Lockley, L. C. (1950). Notes on the history of marketing research. Journal of Marketing, 14(5), 733–736. https://doi.org/10.1177/002224295001400508

March, J. G. (1991). Exploration and exploitation in organizational learning. Organization Science, 2(1), 71–87. https://doi.org/10.1287/orsc.2.1.71

Menon, A., & Varadarajan, P. R. (1992). A model of marketing knowledge use within firms. Journal of Marketing, 56(4), 53–71. https://doi.org/10.1177/002224299205600405

Moorman, C. (1995). Organizational market information processes: Cultural antecedents and new product outcomes. Journal of Marketing Research, 32(3), 318–335. https://doi.org/10.1177/002224379503200307

Narver, J. C., & Slater, S. F. (1990). The effect of a market orientation on business profitability. Journal of Marketing, 54(4), 20–35. https://doi.org/10.1177/002224299005400403

Nonaka, I. (1994). A dynamic theory of organizational knowledge creation. Organization Science, 5(1), 14–37. https://doi.org/10.1287/orsc.5.1.14

Said, E., Macdonald, E. K., Wilson, H. N., & Marcos, J. (2015). How organisations generate and use customer insight. Journal of Marketing Management, 31(9–10), 1158–1179. https://doi.org/10.1080/0267257X.2015.1037785

Sinkula, J. M. (1994). Market information processing and organizational learning. Journal of Marketing, 58(1), 35–45. https://doi.org/10.1177/002224299405800103

Slater, S. F., & Narver, J. C. (1995). Market orientation and the learning organization. Journal of Marketing, 59(3), 63–74. https://doi.org/10.1177/002224299505900306

Szulanski, G. (1996). Exploring internal stickiness: Impediments to the transfer of best practice within the firm. Strategic Management Journal, 17(S2), 27–43. https://doi.org/10.1002/smj.4250171105

Verhoef, P. C., & Leeflang, P. S. H. (2009). Understanding the marketing department's influence within the firm. Journal of Marketing, 73(2), 14–37. https://doi.org/10.1509/jmkg.73.2.14

Wedel, M., & Kannan, P. K. (2016). Marketing analytics for data-rich environments. Journal of Marketing, 80(6), 97–121. https://doi.org/10.1509/jm.15.0413

Weiss, C. H. (1979). The many meanings of research utilization. Public Administration Review, 39(5), 426–431. https://doi.org/10.2307/3109916