The Generation of Knowledge · Knowledge production
How New Knowledge Is Created: From Data to Wisdom
The processes by which people and organisations turn observations into data, data into information, information into knowledge and knowledge into wise action, including induction, abduction, testing and sensemaking, and the places where each step can go wrong.
New knowledge does not appear fully formed. It is made through a sequence of activities: people notice something, record it, organise what they have recorded, interpret it, test their interpretations and decide what to do. Writers on information management often summarise the results of these activities as a hierarchy of data, information, knowledge and wisdom, a framework usually traced to Ackoff (1989). That hierarchy describes what is produced at each level. This article looks at how each transition happens: the human and organisational processes that turn raw observations into understanding and, at best, into good judgement.
From the world to data
Data begin with observation. A survey respondent ticks a box, a sensor records a temperature, a shop till logs a sale. Each of these acts selects a small part of the world and records it in a fixed form. The selection is never neutral. Someone decided which questions to ask, which quantities to measure and which events to log, and those decisions determine what can later be known.
Philosophers of science have long stressed that observation is shaped by prior ideas. Hanson (1958) argued that observation is "theory-laden": what scientists see depends on the concepts and expectations they bring to the observation. Popper (1959) put the point more strongly, arguing that observation always begins from a problem or a point of view, never from a blank mind. For practical research, the lesson is that data collection is the first act of interpretation. A poorly chosen question or a badly designed measure cannot be rescued by later analysis.
From data to information
Data become information when they are organised so that they answer a question. Davenport and Prusak (1998) described several ways in which this happens: data are contextualised (we know what they were collected for), categorised (we know the units of analysis), calculated (they have been analysed mathematically or statistically), corrected (errors have been removed) and condensed (they have been summarised). Each of these is a choice that adds meaning and can also introduce error.
Statistical analysis plays a central role at this stage. Tukey (1977) urged analysts to begin with exploratory data analysis, looking at data in many ways, through plots, summaries and transformations, before testing formal hypotheses. Exploration reveals patterns, outliers and errors that a single pre-planned test might miss. It is also the stage at which surprises are most likely to appear, and surprises are often where new knowledge begins.
From information to knowledge
Information becomes knowledge when it is interpreted, connected to other things that are known and used to explain or predict. This is the most creative step, and philosophers have identified several kinds of reasoning involved.
Induction
Induction generalises from observed cases to a wider pattern. If customers in many samples prefer one design, we infer that customers in general probably prefer it. Induction is the basis of statistical inference, but it is never certain, because the next case may differ from those we have seen.
Abduction
Abduction, a term introduced by the philosopher Charles Sanders Peirce, is reasoning to the best explanation. Faced with a surprising fact, we propose a hypothesis that would make the fact unsurprising (Peirce, 1934). If sales fall in one region only, we might hypothesise that a competitor has entered that market. Abduction is how new ideas enter inquiry; it generates hypotheses that can then be tested.
Testing
New ideas must then be tested. Popper (1959) argued that scientific knowledge grows by proposing bold conjectures and trying hard to refute them. A hypothesis that survives serious attempts at refutation is not proven, but it earns provisional acceptance. In applied research, testing may take the form of a follow-up survey, an experiment, or a check against independent data.
Building theory from data
Some researchers build explanations directly from close study of data. Glaser and Strauss (1967) developed grounded theory, in which categories and explanations emerge from systematic comparison of cases, such as interview transcripts, rather than being specified in advance. This approach is especially useful when little is known about a topic and existing theories may not fit.
Making knowledge shared
In organisations, knowledge must also be shared to be useful. Nonaka (1994) argued that organisational knowledge is created through a continuous conversion between tacit knowledge, held in people's skills and experience, and explicit knowledge, expressed in documents and data. Boisot (1998) described a similar cycle in which knowledge is codified, abstracted, shared and then absorbed back into practice. A finding that stays in one analyst's head, or in an unread report, has not yet become organisational knowledge.
Sensemaking
Much real knowledge creation does not follow a tidy sequence. Weick (1995) described sensemaking as the ongoing process by which people give meaning to confusing or ambiguous situations. Sensemaking is retrospective (we make sense of what has already happened), social (we make sense with others) and driven by plausibility rather than accuracy: people settle on an account that is good enough to act on. Klein et al. (2006) proposed that sensemaking works by fitting data into a frame, an explanatory story, and then either elaborating the frame or questioning and replacing it when the data do not fit.
This view explains why the same data can lead different people to different conclusions. It also shows the value of deliberately seeking data that do not fit the current frame, since that is how frames are corrected.
From knowledge to wisdom
The final level, wisdom, is the hardest to define. Ackoff (1989) associated it with judgement about value: knowing which ends are worth pursuing, not only how to achieve them. Psychologists have studied wisdom empirically. Baltes and Staudinger (2000) defined it as expert knowledge about the fundamental questions of life, including an awareness of uncertainty and of the different values held by different people. Sternberg (1998) proposed a balance theory in which wisdom is the use of knowledge to achieve a common good by balancing the interests of oneself, others and wider society over the short and long term.
For organisations, wisdom means using knowledge with judgement: recognising its limits, weighing the interests of everyone affected and acting in ways that remain defensible as circumstances change.
Where the process goes wrong
Each transition has its own typical failures. At the observation stage, the wrong things are measured, or the right things are measured badly, for example through leading survey questions or unrepresentative samples. At the analysis stage, errors in cleaning and coding pass unnoticed, or analysts test many patterns and report only the striking ones. At the interpretation stage, people accept the first plausible explanation and stop looking, a tendency that Weick's (1995) account of sensemaking predicts. At the sharing stage, findings are simplified until their uncertainty disappears. At the stage of judgement, decision-makers use the findings to support a choice they had already made. Knowing where these failures occur makes it easier to build checks against them into each stage of a project.
Criticisms of the hierarchy
The data-to-wisdom sequence is a useful simplification, but it has been criticised. Rowley (2007) reviewed textbook definitions and found little agreement on what distinguishes each level or on how one is transformed into the next. Frické (2009) argued that the hierarchy wrongly suggests that knowledge can be built up from data by mechanical processing alone, when in fact theory and prior knowledge shape what counts as data in the first place. Both critiques support the point made above: the processes run in both directions, and existing knowledge guides each act of observation.
Practical lessons
For organisations that generate knowledge, the processes described here suggest several practices:
- Design data collection around the question. What is not recorded cannot be known later.
- Explore before confirming. Look at the data broadly before running the planned tests.
- Name the hypotheses. Make explicit the explanations being considered so they can be tested.
- Seek disconfirming evidence. Actively look for data that would show the current account is wrong.
- Share and record what was learned. Convert individual insight into documented, accessible knowledge.
- Judge in context. Consider who will be affected by decisions based on the findings, and how certain the findings really are.
Conclusion
Knowledge creation is a chain of human judgements, from what to observe to how to act. Each link can be strengthened by sound method and weakened by haste or bias. Understanding the processes behind the familiar hierarchy of data, information, knowledge and wisdom helps researchers and their clients see where value is added, and where errors are most likely to enter.
References
Ackoff, R. L. (1989). From data to wisdom. Journal of Applied Systems Analysis, 16, 3–9.
Baltes, P. B., & Staudinger, U. M. (2000). Wisdom: A metaheuristic (pragmatic) to orchestrate mind and virtue toward excellence. American Psychologist, 55(1), 122–136. https://doi.org/10.1037/0003-066X.55.1.122
Boisot, M. H. (1998). Knowledge assets: Securing competitive advantage in the information economy. Oxford University Press.
Davenport, T. H., & Prusak, L. (1998). Working knowledge: How organizations manage what they know. Harvard Business School Press.
Frické, M. (2009). The knowledge pyramid: A critique of the DIKW hierarchy. Journal of Information Science, 35(2), 131–142. https://doi.org/10.1177/0165551508094050
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine.
Hanson, N. R. (1958). Patterns of discovery: An inquiry into the conceptual foundations of science. Cambridge University Press.
Klein, G., Moon, B., & Hoffman, R. R. (2006). Making sense of sensemaking 2: A macrocognitive model. IEEE Intelligent Systems, 21(5), 88–92. https://doi.org/10.1109/MIS.2006.100
Nonaka, I. (1994). A dynamic theory of organizational knowledge creation. Organization Science, 5(1), 14–37. https://doi.org/10.1287/orsc.5.1.14
Peirce, C. S. (1934). Collected papers of Charles Sanders Peirce: Vol. 5. Pragmatism and pragmaticism (C. Hartshorne & P. Weiss, Eds.). Harvard University Press.
Popper, K. R. (1959). The logic of scientific discovery. Hutchinson.
Rowley, J. (2007). The wisdom hierarchy: Representations of the DIKW hierarchy. Journal of Information Science, 33(2), 163–180. https://doi.org/10.1177/0165551506070706
Sternberg, R. J. (1998). A balance theory of wisdom. Review of General Psychology, 2(4), 347–365. https://doi.org/10.1037/1089-2680.2.4.347
Tukey, J. W. (1977). Exploratory data analysis. Addison-Wesley.
Weick, K. E. (1995). Sensemaking in organizations. SAGE.