Skip to content

Inquiry · Philosophy of inquiry

Inquiry into Inquiry: Meta-Level Reflection on Knowledge Generation Processes

Inquiry can be turned on itself: how good are the processes by which we generate knowledge? This article explains meta-level reflection, from individual metacognition to organisational reviews and the science of science, and what the evidence says about its benefits.

ModQu research article · 1,134 words · 13 references (APA 7th edition)

Download PDF

Every inquiry has two levels. At the object level, it asks a question about the world: why are customers leaving, does this drug work, what caused the accident? At the meta level, it can ask questions about itself: was the question well chosen, did the method fit, how reliable are the conclusions, and what should we do differently next time? Inquiry into inquiry is the practice of asking these second-level questions systematically. It happens in individual minds, in teams and organisations, and across whole fields of research, and at every scale the evidence suggests it pays off.

Two levels of questioning

Figure 1 sets the two levels side by side. The object-level question asks what is true; the meta-level question asks how we know and how well we are finding out.

Table pairing object-level questions with meta-level questions for five elements of an inquiry: the question, the method, the evidence, the conclusion and the process.
Figure 1. Object-level and meta-level questions for each element of an inquiry. Diagram by ModQu.

In the individual mind: metacognition

The psychologist John Flavell (1979) introduced the term metacognition for knowledge about and regulation of one's own thinking: knowing what one knows and does not know, choosing strategies, monitoring progress and checking results. Students who monitor their own understanding learn more effectively, and experts in many fields differ from novices partly in the quality of this self-monitoring. Metacognition is the individual form of inquiry into inquiry. It is what happens when a researcher stops and asks, "Am I sure this analysis answers the question I was asked?"

In teams and organisations: reviews and double-loop learning

Chris Argyris and Donald Schön (1978) distinguished single-loop learning, which corrects errors within existing goals and methods, from double-loop learning, which questions the goals and methods themselves. A research team that fixes a coding error is learning in a single loop; a team that asks whether its whole approach to measuring satisfaction is sound is learning in a double loop. Schön (1983) also described how skilled professionals "reflect in action", noticing when a situation surprises them and rethinking their approach on the spot.

The most common organisational tool for meta-level reflection is the after-action review, or debrief: a structured discussion after a project or event of what was intended, what happened, why and what should change. The evidence for such reviews is strong. A meta-analysis by Tannenbaum and Cerasoli (2013) found that well-conducted debriefs improved team and individual performance by about 25% on average. Keiser and Arthur (2021), in a larger meta-analysis, confirmed a substantial positive effect and found it was greater when reviews were facilitated and structured. Ellis and Davidi (2005) showed that reviewing successes as well as failures improves learning, because success also contains lessons that are lost if only failures are examined.

Across a field: meta-research

At the largest scale, inquiry into inquiry has become a research field in its own right, often called meta-research or the science of science. Ioannidis et al. (2015) describe it as the study of research itself: its methods, reporting, reproducibility, evaluation and incentives. Meta-research has produced some of the most consequential findings of recent decades. Ioannidis (2005) argued on statistical grounds that most published findings in some fields were likely to be false. Simmons et al. (2011) showed how flexible choices in data analysis, each defensible on its own, could make almost any result appear statistically significant. The Open Science Collaboration (2015) attempted to replicate 100 psychology studies and found that far fewer produced significant results the second time.

These findings led directly to reforms, including pre-registration of study plans, sharing of data and code, and journals that accept papers before results are known. Munafò et al. (2017) set out a "manifesto for reproducible science" drawing these reforms together. In effect, the research community turned its own methods on itself and changed its practices as a result.

Why reflection is harder than it looks

Looking back on an inquiry is less straightforward than it sounds, because two well-documented biases distort the view. Baruch Fischhoff (1975) showed that once people know how events turned out, they overestimate how predictable the outcome was. This hindsight bias makes past mistakes look obvious and past successes look inevitable, so reviews tend to conclude that the team "should have known". Baron and Hershey (1988) demonstrated outcome bias: people judge the quality of a decision by its result, rating the same reasoning as better when it happened to turn out well. A review that asks only "did it work?" therefore learns the wrong lessons. A good process can produce a bad result through bad luck, and a poor process can be rewarded by good fortune.

The remedy is to review the process against what was knowable at the time. What information was available, what alternatives were considered, how was uncertainty handled, and were the decisions reasonable given those conditions? Records made during the inquiry are essential for this, because memory is reshaped by the outcome.

An example

A market research team forecasts that a new product will reach 8% market share within a year; it reaches 3%. An outcome-focused review would conclude simply that the forecast was wrong. A meta-level review asks how the forecast was produced. It finds that the estimate relied on stated purchase intentions from a survey, without the downward adjustment that experience suggests such intentions need, and that nobody had compared the team's earlier forecasts with actual results. The lesson is not "be more pessimistic next time" but "calibrate intention data against past outcomes and track forecast accuracy routinely". That lesson improves every future forecast, not just the next one.

How to practise inquiry into inquiry

Some practical habits bring the meta level into everyday research:

None of these habits takes much time. A debrief can be completed in half an hour; a decision log can be a few lines in a shared document. What they require is the willingness to treat the organisation's own research process as something that can be studied and improved, in the same way that it studies its customers and markets.

Inquiry that never examines itself repeats its mistakes. Inquiry that does becomes, over time, better at finding things out.

References

Argyris, C., & Schön, D. A. (1978). Organizational learning: A theory of action perspective. Addison-Wesley.

Baron, J., & Hershey, J. C. (1988). Outcome bias in decision evaluation. Journal of Personality and Social Psychology, 54(4), 569–579. https://doi.org/10.1037/0022-3514.54.4.569

Ellis, S., & Davidi, I. (2005). After-event reviews: Drawing lessons from successful and failed experience. Journal of Applied Psychology, 90(5), 857–871. https://doi.org/10.1037/0021-9010.90.5.857

Fischhoff, B. (1975). Hindsight ≠ foresight: The effect of outcome knowledge on judgment under uncertainty. Journal of Experimental Psychology: Human Perception and Performance, 1(3), 288–299. https://doi.org/10.1037/0096-1523.1.3.288

Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist, 34(10), 906–911. https://doi.org/10.1037/0003-066X.34.10.906

Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), Article e124. https://doi.org/10.1371/journal.pmed.0020124

Ioannidis, J. P. A., Fanelli, D., Dunne, D. D., & Goodman, S. N. (2015). Meta-research: Evaluation and improvement of research methods and practices. PLoS Biology, 13(10), Article e1002264. https://doi.org/10.1371/journal.pbio.1002264

Keiser, N. L., & Arthur, W., Jr. (2021). A meta-analysis of the effectiveness of the after-action review (or debrief) and factors that influence its effectiveness. Journal of Applied Psychology, 106(7), 1007–1032. https://doi.org/10.1037/apl0000821

Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, Article 0021. https://doi.org/10.1038/s41562-016-0021

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716

Schön, D. A. (1983). The reflective practitioner: How professionals think in action. Basic Books.

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632

Tannenbaum, S. I., & Cerasoli, C. P. (2013). Do team and individual debriefs enhance performance? A meta-analysis. Human Factors, 55(1), 231–245. https://doi.org/10.1177/0018720812448394