Skip to content

The Generation of Knowledge · Knowledge production

How New Knowledge Is Created: From Data to Wisdom

The processes by which people and organisations turn observations into data, data into information, information into knowledge and knowledge into wise action, including induction, abduction, testing and sensemaking, and the places where each step can go wrong.

ModQu research article · 1,445 words · 15 references (APA 7th edition)

Download PDF

New knowledge does not appear fully formed. It is made through a sequence of activities: people notice something, record it, organise what they have recorded, interpret it, test their interpretations and decide what to do. Writers on information management often summarise the results of these activities as a hierarchy of data, information, knowledge and wisdom, a framework usually traced to Ackoff (1989). That hierarchy describes what is produced at each level. This article looks at how each transition happens: the human and organisational processes that turn raw observations into understanding and, at best, into good judgement.

From the world to data

Data begin with observation. A survey respondent ticks a box, a sensor records a temperature, a shop till logs a sale. Each of these acts selects a small part of the world and records it in a fixed form. The selection is never neutral. Someone decided which questions to ask, which quantities to measure and which events to log, and those decisions determine what can later be known.

Philosophers of science have long stressed that observation is shaped by prior ideas. Hanson (1958) argued that observation is "theory-laden": what scientists see depends on the concepts and expectations they bring to the observation. Popper (1959) put the point more strongly, arguing that observation always begins from a problem or a point of view, never from a blank mind. For practical research, the lesson is that data collection is the first act of interpretation. A poorly chosen question or a badly designed measure cannot be rescued by later analysis.

From data to information

Data become information when they are organised so that they answer a question. Davenport and Prusak (1998) described several ways in which this happens: data are contextualised (we know what they were collected for), categorised (we know the units of analysis), calculated (they have been analysed mathematically or statistically), corrected (errors have been removed) and condensed (they have been summarised). Each of these is a choice that adds meaning and can also introduce error.

Statistical analysis plays a central role at this stage. Tukey (1977) urged analysts to begin with exploratory data analysis, looking at data in many ways, through plots, summaries and transformations, before testing formal hypotheses. Exploration reveals patterns, outliers and errors that a single pre-planned test might miss. It is also the stage at which surprises are most likely to appear, and surprises are often where new knowledge begins.

From information to knowledge

Information becomes knowledge when it is interpreted, connected to other things that are known and used to explain or predict. This is the most creative step, and philosophers have identified several kinds of reasoning involved.

Induction

Induction generalises from observed cases to a wider pattern. If customers in many samples prefer one design, we infer that customers in general probably prefer it. Induction is the basis of statistical inference, but it is never certain, because the next case may differ from those we have seen.

Abduction

Abduction, a term introduced by the philosopher Charles Sanders Peirce, is reasoning to the best explanation. Faced with a surprising fact, we propose a hypothesis that would make the fact unsurprising (Peirce, 1934). If sales fall in one region only, we might hypothesise that a competitor has entered that market. Abduction is how new ideas enter inquiry; it generates hypotheses that can then be tested.

Testing

New ideas must then be tested. Popper (1959) argued that scientific knowledge grows by proposing bold conjectures and trying hard to refute them. A hypothesis that survives serious attempts at refutation is not proven, but it earns provisional acceptance. In applied research, testing may take the form of a follow-up survey, an experiment, or a check against independent data.

Building theory from data

Some researchers build explanations directly from close study of data. Glaser and Strauss (1967) developed grounded theory, in which categories and explanations emerge from systematic comparison of cases, such as interview transcripts, rather than being specified in advance. This approach is especially useful when little is known about a topic and existing theories may not fit.

Making knowledge shared

In organisations, knowledge must also be shared to be useful. Nonaka (1994) argued that organisational knowledge is created through a continuous conversion between tacit knowledge, held in people's skills and experience, and explicit knowledge, expressed in documents and data. Boisot (1998) described a similar cycle in which knowledge is codified, abstracted, shared and then absorbed back into practice. A finding that stays in one analyst's head, or in an unread report, has not yet become organisational knowledge.

Flow of five steps: observe and record, producing data; organise and analyse, producing information; interpret and test, producing knowledge; share and record, producing shared knowledge; judge and act, drawing on wisdom; and a loop showing that the results of action are observed again.
Figure 1. The processes that create each level of understanding, with what each step produces in italics. Based on Ackoff (1989), Davenport and Prusak (1998) and Nonaka (1994); diagram by ModQu.

Sensemaking

Much real knowledge creation does not follow a tidy sequence. Weick (1995) described sensemaking as the ongoing process by which people give meaning to confusing or ambiguous situations. Sensemaking is retrospective (we make sense of what has already happened), social (we make sense with others) and driven by plausibility rather than accuracy: people settle on an account that is good enough to act on. Klein et al. (2006) proposed that sensemaking works by fitting data into a frame, an explanatory story, and then either elaborating the frame or questioning and replacing it when the data do not fit.

This view explains why the same data can lead different people to different conclusions. It also shows the value of deliberately seeking data that do not fit the current frame, since that is how frames are corrected.

From knowledge to wisdom

The final level, wisdom, is the hardest to define. Ackoff (1989) associated it with judgement about value: knowing which ends are worth pursuing, not only how to achieve them. Psychologists have studied wisdom empirically. Baltes and Staudinger (2000) defined it as expert knowledge about the fundamental questions of life, including an awareness of uncertainty and of the different values held by different people. Sternberg (1998) proposed a balance theory in which wisdom is the use of knowledge to achieve a common good by balancing the interests of oneself, others and wider society over the short and long term.

For organisations, wisdom means using knowledge with judgement: recognising its limits, weighing the interests of everyone affected and acting in ways that remain defensible as circumstances change.

Where the process goes wrong

Each transition has its own typical failures. At the observation stage, the wrong things are measured, or the right things are measured badly, for example through leading survey questions or unrepresentative samples. At the analysis stage, errors in cleaning and coding pass unnoticed, or analysts test many patterns and report only the striking ones. At the interpretation stage, people accept the first plausible explanation and stop looking, a tendency that Weick's (1995) account of sensemaking predicts. At the sharing stage, findings are simplified until their uncertainty disappears. At the stage of judgement, decision-makers use the findings to support a choice they had already made. Knowing where these failures occur makes it easier to build checks against them into each stage of a project.

Criticisms of the hierarchy

The data-to-wisdom sequence is a useful simplification, but it has been criticised. Rowley (2007) reviewed textbook definitions and found little agreement on what distinguishes each level or on how one is transformed into the next. Frické (2009) argued that the hierarchy wrongly suggests that knowledge can be built up from data by mechanical processing alone, when in fact theory and prior knowledge shape what counts as data in the first place. Both critiques support the point made above: the processes run in both directions, and existing knowledge guides each act of observation.

Practical lessons

For organisations that generate knowledge, the processes described here suggest several practices:

Conclusion

Knowledge creation is a chain of human judgements, from what to observe to how to act. Each link can be strengthened by sound method and weakened by haste or bias. Understanding the processes behind the familiar hierarchy of data, information, knowledge and wisdom helps researchers and their clients see where value is added, and where errors are most likely to enter.

References

Ackoff, R. L. (1989). From data to wisdom. Journal of Applied Systems Analysis, 16, 3–9.

Baltes, P. B., & Staudinger, U. M. (2000). Wisdom: A metaheuristic (pragmatic) to orchestrate mind and virtue toward excellence. American Psychologist, 55(1), 122–136. https://doi.org/10.1037/0003-066X.55.1.122

Boisot, M. H. (1998). Knowledge assets: Securing competitive advantage in the information economy. Oxford University Press.

Davenport, T. H., & Prusak, L. (1998). Working knowledge: How organizations manage what they know. Harvard Business School Press.

Frické, M. (2009). The knowledge pyramid: A critique of the DIKW hierarchy. Journal of Information Science, 35(2), 131–142. https://doi.org/10.1177/0165551508094050

Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine.

Hanson, N. R. (1958). Patterns of discovery: An inquiry into the conceptual foundations of science. Cambridge University Press.

Klein, G., Moon, B., & Hoffman, R. R. (2006). Making sense of sensemaking 2: A macrocognitive model. IEEE Intelligent Systems, 21(5), 88–92. https://doi.org/10.1109/MIS.2006.100

Nonaka, I. (1994). A dynamic theory of organizational knowledge creation. Organization Science, 5(1), 14–37. https://doi.org/10.1287/orsc.5.1.14

Peirce, C. S. (1934). Collected papers of Charles Sanders Peirce: Vol. 5. Pragmatism and pragmaticism (C. Hartshorne & P. Weiss, Eds.). Harvard University Press.

Popper, K. R. (1959). The logic of scientific discovery. Hutchinson.

Rowley, J. (2007). The wisdom hierarchy: Representations of the DIKW hierarchy. Journal of Information Science, 33(2), 163–180. https://doi.org/10.1177/0165551506070706

Sternberg, R. J. (1998). A balance theory of wisdom. Review of General Psychology, 2(4), 347–365. https://doi.org/10.1037/1089-2680.2.4.347

Tukey, J. W. (1977). Exploratory data analysis. Addison-Wesley.

Weick, K. E. (1995). Sensemaking in organizations. SAGE.