Insights · The insight function
Agile Insights: Faster Research for Faster Decisions
How insight teams adapt research to faster decision cycles, the principles borrowed from agile development and lean experimentation, the evidence on how small and fast studies can be, and the safeguards that keep speed from undermining quality.
Decision cycles in many organisations have become shorter. Products are released in frequent small updates rather than occasional large launches, marketing campaigns are adjusted weekly or daily, and competitors respond quickly to each other's moves. Traditional market research, with its long briefing, design, fieldwork, analysis and reporting stages, often cannot keep up. By the time a three-month study reports, the decision it was meant to inform may already have been made.
"Agile insights" is the name commonly given to the response: approaches that deliver research in shorter cycles, in smaller pieces, and in closer collaboration with the teams who use it. This article explains where these approaches come from, what they involve, what research says about how small and fast studies can safely be, and the safeguards that keep speed from undermining quality.
Where agile ideas come from
The ideas behind agile insights come from product development and software engineering. Takeuchi and Nonaka (1986) described how leading companies had replaced the sequential "relay race" approach to new product development, in which each specialist group completed its stage before passing the work on, with a "rugby" approach in which a cross-functional team worked through overlapping phases together. Their article introduced the term scrum to management, and it influenced later software methods.
In 2001, a group of software developers published the Manifesto for Agile Software Development, which valued individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan (Beck et al., 2001). Agile methods such as Scrum organise work in short, fixed periods, usually called sprints, each ending with something usable and a review of what was learned. Dingsøyr et al. (2012) reviewed a decade of research on agile methods and described how they had spread from small teams to larger organisations, while noting that the theoretical understanding of why they work remained limited. Rigby et al. (2016) described how agile methods had spread beyond software into functions such as marketing and operations, and cautioned that they suit some kinds of work much better than others.
A parallel influence came from entrepreneurship. Ries (2011) proposed the lean startup approach, in which a new venture is treated as a series of experiments: build a minimal version of a product, measure how customers respond, and learn whether to continue or change direction. Blank (2013) described how this emphasis on rapid testing with customers, rather than detailed upfront planning, had spread from startups to established companies. Design thinking, as described by Brown (2008), similarly emphasises early and repeated contact with users and rapid prototyping.
What these approaches share is a view of knowledge as something gained in small increments through repeated contact with reality, rather than in one large study before action begins. For insight work, this implies research that is more frequent, more iterative and more closely tied to the decisions of the teams it serves.
Why speed matters for insight
The value of research depends on its timing. Information that arrives after a decision has been made has no value for that decision. Information value theory makes this explicit: the value of information is the improvement it makes to a decision, so it falls to zero once the decision can no longer change (Howard, 1966).
Speed also matters for organisational responsiveness. Kohli and Jaworski (1990) defined market orientation as including not only the generation and dissemination of market intelligence but also the organisation's responsiveness to it. Moorman and Miner (1998) studied improvisation in new product development, in which planning and execution converge in time, and found that its effects depended on the organisation's memory and the information available in real time. Eisenhardt (1989) studied strategic decision making in fast-moving industries and found that fast decision makers used more information, not less, and in particular relied on real-time information about their operations and environment rather than on forecasts. Speed and evidence are therefore not opposites. Organisations that decide quickly and well tend to have evidence available when they need it.
What agile insights involves
Agile insights is not a single method but a set of practices. The most common are described below.
Short research cycles
Research is organised in short cycles, often of one or two weeks, each designed to answer a specific question that the product or marketing team needs answered now. Instead of one large study, a team runs a series of small ones, each building on the last. The questions for each cycle are agreed with the decision makers, and the results are shared at the end of the cycle, often in a short working session rather than a formal report.
Embedded researchers
Rather than receiving briefs from a distance, researchers work as members of product or marketing teams, attending their planning meetings and reviews. This follows the agile emphasis on collaboration and reduces the time spent on briefing and handover. It also helps researchers understand the decisions their work informs.
Iterative qualitative research
Small rounds of qualitative research, such as interviews or usability tests with five to eight participants, are run repeatedly as a design develops. Each round identifies the most important problems, which are fixed before the next round tests the improved version.
Rapid quantitative research
Online panels and survey platforms allow short surveys to be designed, fielded and analysed within days or even hours. Callegaro et al. (2014) reviewed research on the quality of online panels and described both their practical advantages and the issues of coverage, self-selection and respondent quality that must be managed when using them.
Continuous experimentation
Where products are digital, changes can be tested directly with customers through controlled experiments. Thomke (2003) argued that the falling cost of experimentation was changing how innovation is managed, allowing organisations to test many more ideas than before. Kohavi et al. (2020) described how organisations run large numbers of online controlled experiments, comparing versions of a product or message among randomly assigned customers to measure the effect of each change.
Thomke (2020) argued that the main obstacles to experimentation at scale are cultural rather than technical: organisations must be willing to let test results overrule managers' intuitions, and to accept that most ideas tested will not improve outcomes. These conditions apply to agile insight work more generally. Fast research has value only if teams are prepared to change course when it shows something unexpected.
Reusing existing knowledge
The fastest research is research that has already been done. Agile insight teams check existing studies, dashboards and repositories before commissioning new work, and often answer a question in hours by synthesising what the organisation already knows.
How small can a study be?
Fast research is usually small research, so a central question is how small a study can be while still being useful. The answer depends on the purpose.
Finding problems
When the aim is to find problems, such as usability issues in a design, small samples can be very effective. Virzi (1992) found that most usability problems, and particularly the most severe ones, were found by the first few participants. Nielsen and Landauer (1993) developed a model in which the proportion of problems found with n participants is 1 − (1 − L)^n, where L is the probability that a single participant reveals a given problem. With an average value of L of about 31% across the projects they studied, five participants would be expected to reveal about 85% of the problems.
This result explains the popularity of small, repeated usability tests. It also has limits. Faulkner (2003) ran repeated samples from a larger usability study and found that, although five-participant tests found about 85% of problems on average, individual sets of five varied widely, and some found little more than half. The model also assumes that problems are equally detectable; when problems are rare or affect only some kinds of users, the detection rate is lower and more participants are needed, as the second line in the chart shows. The practical lesson is that small tests are excellent for finding the most common problems, especially when repeated, but not for establishing that a design has no problems.
Understanding experiences and motives
For qualitative interviews aimed at understanding experiences and motives, the relevant concept is saturation: the point at which further interviews add little that is new. Guest et al. (2006) analysed a set of sixty interviews and found that most themes had emerged within the first twelve, with the main elements visible within the first six. Their participants were relatively homogeneous, and they noted that more diverse groups or broader questions would need more interviews. Small rounds of interviews can therefore reveal the main themes quickly, but only for a well-defined group and question.
Measuring and comparing
When the aim is to measure, for example to estimate what share of customers hold a view or to compare two options, small samples are much less forgiving. Estimates from small samples are imprecise, and differences between small groups are often due to chance. A survey of 100 respondents has a margin of error of roughly ten percentage points for a proportion near 50%, too wide to distinguish many options of practical interest. Rapid quantitative research can be fast, but it cannot be made small without losing the precision that measurement requires.
The general principle is that qualitative and problem-finding research can be made both small and fast; measurement research can be made fast but not small.
The risks of speed
Agile insight practices bring real benefits, but they also introduce risks that need to be managed.
Peeking and stopping early
When results arrive continuously, it is tempting to look at them repeatedly and stop as soon as a difference appears. Armitage et al. (1969) showed that repeatedly testing accumulating data for significance, and stopping when a significant result appears, greatly increases the chance of a false positive above the nominal level. Johari et al. (2017) showed that the same problem affects online A/B tests when users monitor results continuously, and described methods that allow valid continuous monitoring. Teams running rapid surveys and experiments should decide sample sizes in advance or use methods designed for sequential monitoring.
Many small studies, many chance findings
Running many small studies multiplies the opportunities for chance findings. Simmons et al. (2011) showed how flexibility in analysis can make false results appear significant, and Gelman and Loken (2014) showed that the problem arises even without deliberate searching, through the many reasonable choices analysts make after seeing data. In an agile setting, where each study is designed and analysed quickly, these choices are rarely documented. Writing down the question and the main measure before each study is a simple safeguard.
Sample quality
Fast online samples bring their own quality issues, including professional respondents, inattentive answering and fraudulent responses. Chandler and Shapiro (2016) reviewed research on crowdsourced convenience samples and described both their usefulness and the checks needed to maintain data quality, such as attention checks and screening for duplicate or inconsistent responses. Callegaro et al. (2014) similarly emphasised that online panels vary greatly in quality and that their results should not be assumed to represent the general population.
Losing the big picture
A series of small studies, each answering a narrow question for one team, can leave the organisation without a broader understanding of its customers. Agile methods suit work that can be divided into small pieces with frequent feedback; Rigby et al. (2016) noted that they are less suited to work requiring long-term, integrated effort. Strategic questions, such as how customer needs are changing or which markets to enter, may require larger, slower and more carefully designed research. An insight function that runs only agile studies may answer many tactical questions while missing strategic ones.
Losing knowledge
Fast studies are often reported informally, in a working session or a short message, and may not be recorded anywhere. Without deliberate effort, their findings are lost when the team moves on. Recording each study briefly in a shared repository preserves the knowledge for future use.
An illustration
Consider a hypothetical subscription app whose product team releases updates every two weeks. Its insight researcher works within the team. At the start of each sprint, the team agrees one or two questions that matter for the next release, such as whether a redesigned sign-up screen is clear, or why new users stop using the app in their first week.
For the sign-up screen, the researcher runs a usability test with six participants on the Tuesday, shares the main problems in a thirty-minute session on the Wednesday, and the designers fix them before the next test the following week. After three rounds, the remaining problems are minor, and the screen is released as a controlled experiment with half of new users. The experiment, with a sample size fixed in advance, shows whether the new screen increases completed sign-ups.
For the question about early drop-off, the researcher first checks the repository and finds an earlier interview study on the same topic. Its findings are a year old, so the researcher runs eight short interviews with recent drop-outs, focusing on whether the earlier themes still apply. Most do, but one new theme appears: confusion about a recently introduced feature. The finding is shared at the sprint review and recorded in the repository, and a larger survey is planned for the strategic track to measure how widespread the confusion is.
In each case, the research is fast because it is small, focused and close to the team. It remains reliable because each study is matched to its purpose: small qualitative rounds to find problems and themes, an experiment with a fixed sample to measure an effect, and a larger survey when the question is one of measurement.
Balancing speed and rigour
A practical way to balance speed and rigour is to match the research approach to the decision. Decisions that are easily reversed and low in stakes, such as the wording of a button or the order of items in a menu, can be made on fast, small studies or directly through experiments. Decisions that are costly to reverse and high in stakes, such as a pricing strategy or a major launch, deserve more careful research even if it takes longer. Information value theory supports this approach: the value of more precise research rises with the stakes of the decision and falls as the decision deadline approaches (Howard, 1966).
Many insight teams therefore operate two speeds: an agile track that serves product and marketing teams with fast, iterative research, and a strategic track that conducts larger studies on questions that shape the organisation's direction. The two tracks share standards, tools and a repository, so that fast studies can draw on strategic research and strategic research can be informed by what fast studies reveal.
Making agile insights work
Several practices help agile insights succeed.
Agree the question and decision first. Every study, however small, should begin with a clear statement of the decision it informs and the question it answers.
Use templates. Standard designs for common questions, such as concept tests, message tests and usability tests, reduce design time and make results comparable across studies.
Set minimum standards. Define minimum sample sizes for measurement, quality checks for online samples, and rules for when results can be shared.
Pre-specify the main measure. Writing down the main measure and the comparison of interest before fieldwork protects against finding patterns that are not there.
Record everything. Store a short record of each study, including its question, sample, findings and limitations, in a shared repository.
Review periodically. Step back regularly to synthesise what a series of studies has shown, identify gaps, and decide whether a larger study is needed.
Embed and collaborate. Place researchers within the teams they serve, while maintaining links to a central insight function that sets standards and sees across teams. Said et al. (2015) found that managers valued insight that was new, shared and acted upon; agile practices make insight more likely to be acted upon, and a central function makes it more likely to be shared.
Key takeaways
Agile insights adapts research to shorter decision cycles through short research sprints, embedded researchers, iterative qualitative research, rapid surveys, continuous experimentation and reuse of existing knowledge. Its ideas come from agile software development, the lean startup and design thinking. Research shows that small studies can be very effective for finding common problems and identifying main themes, but measurement requires adequate samples regardless of speed. The main risks are early stopping, chance findings from many small studies, poor sample quality, loss of the strategic picture and loss of knowledge. Matching the research approach to the stakes and reversibility of each decision, and maintaining shared standards and records, allows organisations to gain speed without sacrificing the reliability of their conclusions.
References
Armitage, P., McPherson, C. K., & Rowe, B. C. (1969). Repeated significance tests on accumulating data. Journal of the Royal Statistical Society: Series A (General), 132(2), 235–244. https://doi.org/10.2307/2343787
Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R. C., Mellor, S., Schwaber, K., Sutherland, J., & Thomas, D. (2001). Manifesto for agile software development. https://agilemanifesto.org/
Blank, S. (2013). Why the lean start-up changes everything. Harvard Business Review, 91(5), 63–72.
Brown, T. (2008). Design thinking. Harvard Business Review, 86(6), 84–92.
Callegaro, M., Baker, R., Bethlehem, J., Göritz, A. S., Krosnick, J. A., & Lavrakas, P. J. (Eds.). (2014). Online panel research: A data quality perspective. Wiley. https://doi.org/10.1002/9781118763520
Chandler, J., & Shapiro, D. (2016). Conducting clinical research using crowdsourced convenience samples. Annual Review of Clinical Psychology, 12, 53–81. https://doi.org/10.1146/annurev-clinpsy-021815-093623
Dingsøyr, T., Nerur, S., Balijepally, V., & Moe, N. B. (2012). A decade of agile methodologies: Towards explaining agile software development. Journal of Systems and Software, 85(6), 1213–1221. https://doi.org/10.1016/j.jss.2012.02.033
Eisenhardt, K. M. (1989). Making fast strategic decisions in high-velocity environments. Academy of Management Journal, 32(3), 543–576. https://doi.org/10.5465/256434
Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers, 35(3), 379–383. https://doi.org/10.3758/BF03195514
Gelman, A., & Loken, E. (2014). The statistical crisis in science. American Scientist, 102(6), 460–465. https://doi.org/10.1511/2014.111.460
Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), 59–82. https://doi.org/10.1177/1525822X05279903
Howard, R. A. (1966). Information value theory. IEEE Transactions on Systems Science and Cybernetics, 2(1), 22–26. https://doi.org/10.1109/TSSC.1966.300074
Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2017). Peeking at A/B tests: Why it matters, and what to do about it. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1517–1525). ACM. https://doi.org/10.1145/3097983.3097992
Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. https://doi.org/10.1017/9781108653985
Kohli, A. K., & Jaworski, B. J. (1990). Market orientation: The construct, research propositions, and managerial implications. Journal of Marketing, 54(2), 1–18. https://doi.org/10.1177/002224299005400201
Moorman, C., & Miner, A. S. (1998). The convergence of planning and execution: Improvisation in new product development. Journal of Marketing, 62(3), 1–20. https://doi.org/10.1177/002224299806200301
Nielsen, J., & Landauer, T. K. (1993). A mathematical model of the finding of usability problems. In Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems (pp. 206–213). ACM. https://doi.org/10.1145/169059.169166
Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation to create radically successful businesses. Crown Business.
Rigby, D. K., Sutherland, J., & Takeuchi, H. (2016). Embracing agile. Harvard Business Review, 94(5), 40–50.
Said, E., Macdonald, E. K., Wilson, H. N., & Marcos, J. (2015). How organisations generate and use customer insight. Journal of Marketing Management, 31(9–10), 1158–1179. https://doi.org/10.1080/0267257X.2015.1037785
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
Takeuchi, H., & Nonaka, I. (1986). The new new product development game. Harvard Business Review, 64(1), 137–146.
Thomke, S. (2020). Building a culture of experimentation. Harvard Business Review, 98(2), 40–47.
Thomke, S. H. (2003). Experimentation matters: Unlocking the potential of new technologies for innovation. Harvard Business School Press.
Virzi, R. A. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34(4), 457–468. https://doi.org/10.1177/001872089203400407