Glossary

ModQu glossary

Definitions of the key terms used across ModQu's articles on surveys, research, statistics, intelligence, insights, inquiry, investigation and the generation of knowledge, with sources.

All terms A–Z

A

A/B test also: split test

A randomised controlled experiment, usually online, in which users are randomly assigned to two or more versions of a product, page or message and an outcome such as conversion is compared. Randomisation makes the versions comparable, so differences in outcome can be attributed to the change. Common pitfalls include stopping a test as soon as results look significant and testing many metrics without correction. (Kohavi et al., 2009)

Research · Research design
See also: Randomised controlled trial, Multiple comparisons, Statistical power
Read more: A/B Testing as Applied Experimental Research

Abduction also: inference to the best explanation

Reasoning to the hypothesis that would best explain a surprising observation. Peirce treated it as the only form of inference that introduces new ideas; contemporary philosophy discusses it as inference to the best explanation, in which explanations are compared on criteria such as scope, simplicity and coherence with what is already known. (Lipton, 2004)

Inquiry · Logic and reasoning
See also: Deduction, Induction, Occam's razor
Read more: Abductive Reasoning: Inference to the Best Explanation · Induction, Deduction and Abduction in Research Reasoning

Absorptive capacity

A firm's ability to recognise the value of new external information, assimilate it and apply it to commercial ends. Cohen and Levinthal argued that it depends largely on prior related knowledge, which is why firms that do their own research are better at using the research of others. (Cohen & Levinthal, 1990)

The Generation of Knowledge · Knowledge management
See also: Knowledge transfer, Double-loop learning
Read more: Absorptive Capacity: How Firms Learn From External Knowledge

Acquiescence bias also: yea-saying

The tendency of some respondents to agree with statements regardless of their content, sometimes called "yea-saying". It inflates agreement on agree–disagree items and can create spurious correlations between items that share the same direction of wording. Common remedies are item-specific question formats (asking directly about the attribute rather than asking for agreement) and balanced scales that mix positively and negatively worded items. (Krosnick, 1999; Tourangeau et al., 2000)

Surveys · Nonresponse, weighting and data quality
See also: Likert scale, Satisficing, Social desirability bias
Read more: Acquiescence Bias: The Tendency to Agree and How to Counter It

Action research

An approach in which researchers and participants study a practical problem while acting to change it, moving through repeated cycles of planning, action, observation and reflection. Kurt Lewin coined the term in the 1940s. It values practical improvement and the knowledge of participants alongside formal findings. (Lewin, 1946)

Research · Research design
See also: Case study
Read more: Action Research: Studying Change While Making It

Actionable insight

An insight specific and credible enough to change a decision: it identifies who is affected, what to do differently and why, and it is backed by evidence that decision-makers trust. Findings that are interesting but do not point to an action, or that the organisation cannot act on, fall short of this standard. (Rousseau, 2006)

Insights · What an insight is
See also: Insight, Finding
Read more: Actionable Insight: Criteria for Insight That Changes Decisions · Data, Findings and Insights: Why the Difference Matters

Admissibility

Whether evidence may be considered by a court or tribunal. Rules vary by jurisdiction but usually require relevance, reliability and lawful collection. For expert scientific testimony in US federal courts, the Daubert standard asks whether a method has been tested, peer reviewed, has a known error rate and is generally accepted. (Daubert v. Merrell Dow Pharmaceuticals, Inc., 1993; National Research Council, 2009)

Investigation · Foundations
See also: Chain of custody, Expert witness, Evidence
Read more: Evidence: Types, Weight and Admissibility · The Role of Expert Witnesses in Translating Investigative Knowledge for Audiences

Aha moment also: eureka moment

The sudden, subjectively surprising realisation of a solution or understanding that accompanies insight problem solving. Neuroscience research links it to a burst of activity in the right anterior temporal lobe shortly before the solution reaches awareness, preceded by a brief reduction in visual input as attention turns inward. (Kounios & Beeman, 2014)

Insights · What an insight is
See also: Insight, Incubation, Restructuring
Read more: The Cognitive Science of Insight: The "Aha" Moment

All-source intelligence also: all-source fusion

Intelligence produced by combining information from every available collection discipline, such as human sources, signals, imagery and open sources, into one assessment. Fusing sources allows analysts to corroborate reports and compensate for the gaps and biases of any single source. (Kent, 1949)

Intelligence · Analysis and tradecraft
See also: Open-source intelligence, Corroboration, Source evaluation
Read more: All-Source Intelligence Fusion Methodologies for Comprehensive Knowledge Integration

Analysis of competing hypotheses also: ACH

A structured analytic technique developed by Richards Heuer in which analysts list all plausible hypotheses, array the evidence against each in a matrix, and focus on disconfirming evidence: the most likely hypothesis is the one with the least evidence against it, not the most evidence for it. It is designed to counter confirmation bias and premature closure. (Heuer, 1999)

Intelligence · Analysis and tradecraft
See also: Structured analytic techniques, Confirmation bias, Key assumptions check
Read more: Analysis of Competing Hypotheses

Anchoring vignettes

Short descriptions of hypothetical people or situations that every respondent rates on the same scale used for their self-assessment. Because the vignettes are fixed, differences in how groups rate them reveal differences in how those groups use the scale, which can then be corrected. The method addresses differential item functioning, the problem that two people with the same true level may choose different answer categories. (King et al., 2004)

Surveys · Questionnaire design
See also: Measurement invariance, Likert scale
Read more: Anchoring Vignettes in Survey Research: Correcting Differential Item Functioning for Better Knowledge · The Use of Vignette Experiments Within Surveys for Contextualized Knowledge

Anti-money laundering also: AML

Laws, controls and investigations designed to stop criminals from disguising the proceeds of crime as legitimate funds. Laundering is usually described in three stages: placement, layering and integration. The Financial Action Task Force recommendations set international standards, including customer due diligence and suspicious transaction reporting. (Financial Action Task Force, 2012)

Investigation · Business and corporate investigation
See also: Due diligence, Forensic accounting, Blockchain analysis
Read more: Anti-Money Laundering Investigations

Apophenia

The tendency to perceive meaningful patterns or connections in random or unrelated data. It may have evolved because the cost of missing a real pattern often exceeded the cost of seeing a false one. In data analysis, large datasets with many variables make spurious patterns easy to find and easy to mistake for insight. (Foster & Kokko, 2009)

Insights · Insight quality and pitfalls
See also: Spurious correlation, Confirmation bias, Multiple comparisons
Read more: The Illusion of Insight: Apophenia and False Knowledge Generation in Big Data

Appreciative inquiry

An approach to organisational change, developed by David Cooperrider and Suresh Srivastva, that asks what works well and builds on it rather than diagnosing problems. It typically moves through discovery, dream, design and destiny phases. (Cooperrider & Srivastva, 1987)

Inquiry · Inquiry in organisations
See also: Double-loop learning
Read more: Appreciative Inquiry: Asking What Works

Argument mapping

Drawing the structure of an argument as a diagram that shows the main claim, the reasons for and objections against it, and how they relate. Studies in university courses found that practice with argument maps improved students' critical-thinking test scores more than conventional teaching. (Twardy, 2004)

Inquiry · Logic and reasoning
See also: Critical thinking, Logical fallacy
Read more: Argument Mapping and the Structure of Reasoning

Attention check also: instructed response item

An item placed in a questionnaire to detect respondents who are not reading the questions, for example an instruction to select a specific answer. Failure rates are used to flag or remove inattentive cases. Evidence suggests such checks identify careless responding but can also irritate attentive respondents, so they are best combined with other indicators such as completion time and response patterns. (Oppenheimer et al., 2009; Meade & Craig, 2012)

Surveys · Nonresponse, weighting and data quality
See also: Careless responding, Satisficing, Straightlining
Read more: Attention Checks and Instructed Response Items: Do They Work? · Detecting Careless and Inattentive Responding in Online Surveys

B

Base rate neglect also: base rate fallacy

The tendency to ignore the prior frequency of an event when judging the probability of a case, relying instead on specific information such as a test result. It leads people to overestimate the chance that a positive test means disease when the disease is rare. Presenting information as natural frequencies (for example, 9 out of 1,000) greatly reduces the error. (Bar-Hillel, 1980; Gigerenzer & Hoffrage, 1995)

Statistics · Communication and misuse
See also: Bayes' theorem, Prior probability
Read more: Base Rate Neglect and the Misreading of Probabilities · Bayesian Reasoning in Everyday Inquiry

Bayes' theorem

A rule for updating the probability of a hypothesis in light of evidence: the posterior probability is proportional to the prior probability times the likelihood of the evidence under the hypothesis. Formally, P(H | E) = P(E | H) × P(H) / P(E). It underlies Bayesian statistics and much of rational belief revision. (Bayes, 1763; Jaynes, 2003)

Statistics · Bayesian and computational
See also: Prior probability, Bayesian inference, Base rate neglect
Read more: Bayes' Theorem and Updating Beliefs With Evidence · Bayesian Reasoning in Everyday Inquiry

Bayesian inference

An approach to statistics in which unknown quantities are treated as having probability distributions. A prior distribution expressing what is known before the data is combined with the likelihood of the data to give a posterior distribution, from which estimates and credible intervals are read directly as probability statements about the parameters. (Edwards et al., 1963; Jaynes, 2003)

Statistics · Bayesian and computational
See also: Bayes' theorem, Frequentist inference, Prior probability
Read more: Frequentist and Bayesian Statistics: Two Philosophies of Probability · Bayesian Inference in Market Research

Behavioural economics

The study of how psychological factors such as limited attention, loss aversion, present bias and social norms make economic choices depart from the predictions of standard rational-choice models. It supplies many consumer insights, for example that defaults strongly shape what people choose. (Tversky & Kahneman, 1974; Kahneman, 2011; Thaler & Sunstein, 2008)

Insights · Generating insight
See also: Nudge, Heuristics and biases, Framing effect
Read more: Behavioural Economics as a Source of Consumer Insight

Blockchain analysis

Tracing cryptocurrency transactions on public ledgers to link addresses to each other and to real-world entities such as exchanges. Although users are pseudonymous, studies of Bitcoin have shown that clustering heuristics can follow flows of funds through many transactions. (Meiklejohn et al., 2013)

Investigation · Digital investigation
See also: Anti-money laundering, Digital forensics
Read more: Blockchain Analysis and Cryptocurrency Investigation

Bloom's taxonomy

A classification of learning objectives by cognitive level. The 2001 revision orders them as remember, understand, apply, analyse, evaluate and create, and is often used to design questions that move from recall to higher-order thinking. (Anderson & Krathwohl, 2001)

Inquiry · Inquiry in education and learning
See also: Critical thinking
Read more: Bloom's Taxonomy and Levels of Questioning

Bootstrap

A resampling method introduced by Bradley Efron in 1979 that estimates the sampling distribution of a statistic by repeatedly drawing samples, with replacement, from the observed data and recomputing the statistic each time. The spread of the resampled values gives standard errors and confidence intervals without strong distributional assumptions. (Efron, 1979)

Statistics · Bayesian and computational
See also: Standard error, Confidence interval, Monte Carlo simulation
Read more: Bootstrap Methods and Resampling

Brier score

A measure of the accuracy of probability forecasts, equal to the mean squared difference between the forecast probabilities and the outcomes (1 if the event happened, 0 if not). Lower is better. It can be decomposed into calibration and resolution, and it was the main scoring rule in the forecasting tournaments that identified superforecasters. (Brier, 1950; Murphy, 1973)

Intelligence · Analysis and tradecraft
See also: Calibration, Superforecasting
Read more: Superforecasting: What Research Says About Accurate Prediction

Business intelligence also: BI

The processes, technologies and practices an organisation uses to collect, integrate, analyse and present data about its operations and environment in support of decisions. Hans Peter Luhn used the term in 1958 for an automated system to disseminate information; it now usually refers to data warehouses, reporting and dashboards. (Luhn, 1958; Chen et al., 2012)

Intelligence · Business intelligence
See also: Key performance indicator, Dashboard, Descriptive, predictive and prescriptive analytics
Read more: Business Intelligence: Systems, Processes and Value · A History of Business Intelligence From Ledgers to Dashboards

C

Calibration

The match between stated probabilities and observed frequencies: a forecaster is well calibrated if events given a 70% chance happen about 70% of the time. Calibration can be assessed with calibration curves and the calibration component of the Brier score, and it improves with training and feedback. (Murphy, 1973; Mandel & Barnes, 2014)

Intelligence · Analysis and tradecraft
See also: Brier score, Words of estimative probability
Read more: Superforecasting: What Research Says About Accurate Prediction · Expressing Probability in Intelligence Estimates: Words of Estimative Probability

Careless responding also: insufficient effort responding

Answering survey items without adequate attention to their content, producing responses that do not reflect the respondent's true state. Indicators include very fast completion, long strings of identical answers, inconsistent answers to items that should agree, and failed attention checks. Careless responding adds noise and can bias estimates and correlations if it is not detected. (Meade & Craig, 2012)

Surveys · Nonresponse, weighting and data quality
See also: Attention check, Straightlining, Satisficing
Read more: Detecting Careless and Inattentive Responding in Online Surveys · Speeders, Straightliners and Bots: Cleaning Online Survey Data

Case study

An in-depth empirical study of a contemporary phenomenon in its real-life context, especially when the boundary between phenomenon and context is unclear. Case studies use multiple sources of evidence and can be single or multiple, exploratory, descriptive or explanatory. They generalise analytically, to theory, rather than statistically, to a population. (Yin, 2018)

Research · Research design
See also: Triangulation
Read more: Case Study Research: Design, Rigour and Generalisation · Case Selection Strategies in Multiple-Case Research Designs

Central limit theorem

The result that the distribution of the mean (or sum) of many independent observations with finite variance approaches a normal distribution as the sample size grows, whatever the shape of the original distribution. It explains why the normal distribution is used so widely for sampling distributions and confidence intervals. (Lohr, 2021)

Statistics · Foundations
See also: Normal distribution, Sampling distribution, Law of large numbers
Read more: The Central Limit Theorem Explained

Chain of custody

The documented, unbroken record of who collected, handled, transferred, stored and analysed an item of evidence, and when. It shows that the evidence presented is the same as that collected and has not been altered. In digital investigations it is supported by hash values that show files have not changed. (Casey, 2011; National Research Council, 2009)

Investigation · Foundations
See also: Admissibility, Digital forensics, Provenance
Read more: The Chain of Custody and the Integrity of Evidence

Chi-square test

A family of tests introduced by Karl Pearson in 1900 that compare observed frequencies with those expected under a hypothesis. The test of independence asks whether two categorical variables in a cross-tabulation are associated. With complex survey samples, design-adjusted versions such as the Rao–Scott correction are needed. (Pearson, 1900)

Statistics · Tests and models
See also: P-value
Read more: Chi-Square Tests for Categorical Survey Data · Significance Testing in Cross-Tabulated Survey Results

Churn also: attrition

The loss of customers over a period, often expressed as a rate. Churn insight combines behavioural data showing who leaves and when with survey and interview data explaining why, so that retention efforts can target the causes rather than the symptoms. (Gupta et al., 2006)

Insights · Customer and market insight
See also: Customer lifetime value, Voice of the customer
Read more: Churn Insight: Why Customers Leave

Citizen science

Scientific research in which members of the public take part in collecting, analysing or interpreting data, for example in bird counts. It can greatly extend the scale of data collection and public understanding of science, with data quality managed through training and validation. (Bonney et al., 2009)

Inquiry · Public and formal inquiry
See also: Deliberative inquiry
Read more: Participatory Research and Citizen Science · Citizen-Generated Knowledge

Cluster analysis

Methods that group cases so that cases in the same cluster are more similar to each other than to those in other clusters, using distances such as Euclidean distance. K-means and hierarchical clustering are common. In market research it is used to build segments, whose usefulness depends on the variables chosen. (Hastie et al., 2009)

Statistics · Tests and models
See also: Segmentation, Latent class analysis
Read more: Cluster Analysis for Market Segmentation

Cluster sampling

A probability design in which the population is divided into groups (clusters), such as schools or city blocks, a sample of clusters is selected, and units are surveyed within the selected clusters. It reduces travel and listing costs but usually lowers precision because units in the same cluster tend to be alike, which raises the design effect. (Kish, 1965; Lohr, 2021)

Surveys · Sampling
See also: Design effect, Probability sample, Stratified sampling
Read more: Probability Sampling Explained: Simple Random, Systematic, Stratified and Cluster · Multistage Sampling Designs for Large Populations

Coding

Labelling segments of qualitative data, such as interview passages, with short descriptive or interpretive codes so that they can be compared, grouped into categories and developed into themes or theory. Codes may come from the data (inductive) or from an existing framework (deductive). (Braun & Clarke, 2006; Glaser & Strauss, 1967)

Research · Qualitative research
See also: Thematic analysis, Grounded theory, Content analysis
Read more: Coding Qualitative Data: Manual and Software-Assisted Approaches

Cognitive interview

An interviewing method for witnesses, developed by Ronald Fisher and Edward Geiselman, based on principles of memory: rapport, mental reinstatement of the context, reporting everything, and recalling events in different orders and from different perspectives. A meta-analysis found that it elicits substantially more correct detail than standard interviews, with a small increase in errors. (Fisher & Geiselman, 1992; Memon et al., 2010)

Investigation · Foundations
See also: PEACE model, Eyewitness testimony
Read more: The Cognitive Interview: Improving Witness Recall · Cognitive Interviewing versus Traditional Interrogation: Knowledge Yield Comparisons

Cognitive interviewing

A pretesting method in which respondents answer draft survey questions while thinking aloud or responding to probes about how they understood each question and arrived at their answer. It reveals problems of comprehension, recall, judgement and response mapping before a survey is fielded. It is qualitative and usually involves small purposive samples. (Willis, 2005; Tourangeau et al., 2000)

Surveys · Questionnaire design
See also: Survey response process
Read more: Cognitive Interviewing as a Tool for Pretesting Questionnaires · Pilot Testing Surveys: Methods for Catching Problems Before Launch

Cohen's kappa

A measure of agreement between two raters assigning cases to categories, corrected for the agreement expected by chance. Kappa is 1 for perfect agreement and 0 for chance-level agreement. It is widely used to check the reliability of coding in content analysis and qualitative research. (Cohen, 1960)

Statistics · Survey statistics
See also: Inter-rater reliability, Reliability
Read more: Inter-Rater Reliability: Cohen's Kappa and Related Measures

Coherentism

The view that a belief is justified by its coherence with a wider system of beliefs, rather than by resting on basic foundations. Laurence BonJour developed an influential version, later revising his position. (BonJour, 1985)

The Generation of Knowledge · Epistemology
See also: Foundationalism
Read more: Foundationalism, Coherentism and the Structure of Knowledge

Collective intelligence

The capacity of a group to perform well across a range of tasks, beyond the abilities of its individual members. Research has found a general collective intelligence factor in small groups that is more strongly related to members' social sensitivity and equality of conversational turn-taking than to the average or maximum intelligence of members. (Woolley et al., 2010)

Intelligence · Artificial and collective intelligence
See also: Wisdom of crowds, Prediction market
Read more: Collective Intelligence and the Wisdom of Crowds · Surveys as Instruments of Collective Intelligence

Collider bias

Bias that arises when an analysis conditions on (selects on, stratifies by or adjusts for) a variable that is caused by both the exposure and the outcome. Conditioning on such a collider creates a spurious association between its causes, as in Berkson's finding that two diseases appear related in hospital samples. (Elwert & Winship, 2014; Berkson, 1946; Hernán et al., 2004)

Statistics · Causal inference
See also: Confounding, Directed acyclic graph, Selection bias
Read more: Correlation Is Not Causation: Spurious Relationships in Data · Directed Acyclic Graphs for Causal Reasoning

Communities of practice

Groups of people who share a concern or a craft and learn how to do it better by interacting regularly. Etienne Wenger described them through mutual engagement, joint enterprise and a shared repertoire, and showed that much organisational knowledge lives and develops in such communities rather than in formal structures. (Wenger, 1998)

The Generation of Knowledge · Knowledge management
See also: Tacit knowledge, Knowledge management
Read more: Communities of Practice

Community of inquiry

A group that pursues understanding together through shared questioning and reasoning. Peirce saw the community of investigators, rather than the individual, as the agent of scientific inquiry. In education the term names a framework of cognitive, social and teaching presence for collaborative online learning. (Peirce, 1877; Garrison et al., 1999)

Inquiry · Philosophy of inquiry
See also: Inquiry, Social epistemology
Read more: Communities of Inquiry

Competitive intelligence also: CI

The legal and ethical collection and analysis of information about competitors, markets and the business environment to support strategic and tactical decisions. It differs from industrial espionage in relying on public sources, interviews and observation rather than deception or theft. (Porter, 1979)

Intelligence · Market and competitive intelligence
See also: Market intelligence, Open-source intelligence, Five forces
Read more: Competitive Intelligence: Ethical Collection and Analysis · The Ethics and Legality of Competitive Intelligence

Confidence interval

A range computed from sample data by a procedure that, over repeated samples, contains the true parameter a stated proportion of the time, such as 95%. A particular interval either contains the parameter or does not; the 95% describes the procedure, not the probability for one interval. Jerzy Neyman set out the theory in 1937. (Neyman, 1937; Greenland et al., 2016)

Statistics · Inference
See also: Margin of error, Standard error, P-value
Read more: Confidence Intervals: Meaning and Misinterpretation

Confirmation bias

The tendency to search for, interpret, favour and remember information in ways that support existing beliefs, and to give less weight to contrary evidence. It is among the most pervasive reasoning errors and affects analysts, investigators and researchers as much as lay people. (Nickerson, 1998)

Insights · Insight quality and pitfalls
See also: Analysis of competing hypotheses, Tunnel vision, Apophenia
Read more: Confirmation Bias and the Search for Expected Insights · Confirmation Bias and Tunnel Vision in Investigations · Cognitive Biases in Intelligence Analysis

Confirmation bias in investigation

The tendency of investigators and forensic examiners to seek, interpret and weigh evidence in ways that confirm an initial suspect or theory. Contextual information, such as knowing a suspect has confessed, can influence even technical judgements like fingerprint comparisons. Blind procedures and sequential unmasking reduce it. (Kassin et al., 2013; Nickerson, 1998)

Investigation · Foundations
See also: Tunnel vision, Confirmation bias
Read more: Confirmation Bias and Tunnel Vision in Investigations

Confounding

Distortion of the estimated effect of an exposure on an outcome because a third variable causes both. Confounding can create, hide or reverse an association. It is controlled by randomisation, or in observational data by adjustment, matching or stratification on the confounders, which requires that they are known and measured. (Pearl, 2009; Hernán et al., 2004)

Statistics · Causal inference
See also: Collider bias, Directed acyclic graph, Simpson's paradox, Spurious correlation
Read more: Confounding Variables and How to Control for Them · Correlation Is Not Causation: Spurious Relationships in Data

Construct validity

The degree to which a measure actually captures the theoretical concept (construct) it is meant to measure. It is established by showing that scores relate to other measures as theory predicts: strongly with measures of the same construct (convergent evidence) and weakly with measures of different constructs (discriminant evidence). Modern validity theory treats it as the unifying form of validity. (Cronbach & Meehl, 1955; Messick, 1995)

Research · Quantitative and mixed methods
See also: Reliability, Operationalisation, Internal validity
Read more: Construct Validity and How to Establish It · Reliability and Validity: The Twin Pillars of Measurement

Content analysis

A method for making replicable and valid inferences from texts, images or other communication by coding their content into categories according to explicit rules. Quantitative content analysis counts coded features; qualitative variants interpret them. Reliability is checked by having independent coders apply the same scheme. (Krippendorff, 2019)

Research · Quantitative and mixed methods
See also: Thematic analysis, Inter-rater reliability
Read more: Content Analysis: Quantifying Text and Media

Continued influence effect

The tendency of misinformation to keep affecting reasoning after it has been clearly corrected, especially when the correction leaves a gap in the explanation that the false information filled. Corrections work best when they offer an alternative account. (Lewandowsky et al., 2012)

Inquiry · Ethics and limits
See also: Prebunking, Disinformation
Read more: Epistemic Responsibility in an Age of Misinformation

Correlation coefficient

A number between −1 and +1 summarising the strength and direction of association between two variables. Pearson's r measures linear association; Spearman's rho measures monotonic association using ranks. A correlation does not show causation and can be distorted by outliers and by non-linear relationships, as Anscombe's quartet illustrates. (Anscombe, 1973)

Statistics · Tests and models
See also: Spurious correlation, Confounding, Regression
Read more: Correlation: Pearson, Spearman and Their Limits · Correlation Is Not Causation: Spurious Relationships in Data

Corroboration

Independent evidence that confirms a claim or another piece of evidence. Corroboration is strong only when the sources are genuinely independent, so that one cannot be an echo of the other, and when the confirming detail could not easily have been known or guessed. (Heuer, 1999)

Investigation · Foundations
See also: Source evaluation, Triangulation
Read more: Corroboration: Testing Whether Independent Sources Agree

Counterfactual

The outcome that would have occurred under a condition other than the one that actually happened. In the potential outcomes framework each unit has an outcome under treatment and under control, and the causal effect is their difference. Because only one is ever observed, causal inference is a problem of estimating the missing counterfactual. (Rubin, 1974; Holland, 1986)

Statistics · Causal inference
See also: Potential outcomes, Randomised controlled trial
Read more: Causal Inference: From Association to Cause · Using Counterfactual Thinking to Improve Interpretation

Counterintelligence

Activities to identify, deter and counter efforts by others to obtain an organisation's protected information or to deceive it. In business it covers protecting trade secrets and strategic plans from competitors, insiders and foreign intelligence services. (Kent, 1949)

Intelligence · Organisational intelligence
See also: Deception
Read more: Counterintelligence for Businesses: Protecting Against Espionage

Coverage error

Error that arises when the sampling frame does not match the target population, because some members are missing from the frame (undercoverage) or the frame includes units outside the population (overcoverage). Bias results when those missing differ from those covered on the variables being measured. It is one of the error sources in the total survey error framework. (Groves et al., 2009; Lohr, 2021)

Surveys · Sampling
See also: Sampling frame, Total survey error
Read more: Sampling Frames and Coverage Error: Who Is Missing From Your List?

Critical realism

A philosophy of science holding that reality exists independently of our knowledge of it, but that our knowledge is fallible and socially produced. It distinguishes the real (structures and mechanisms), the actual (events) and the empirical (what is observed), and directs research to the mechanisms that produce outcomes in particular contexts. Realist evaluation draws on it. (Pawson & Tilley, 1997)

Research · Philosophy and paradigms
See also: Realist evaluation, Positivism, Interpretivism
Read more: Critical Realist Approaches to Research and Stratified Knowledge Claims

Critical thinking

Purposeful, self-regulatory judgement that results in interpretation, analysis, evaluation and inference, together with explanation of the evidence and reasoning behind it, according to the consensus definition produced by a panel of 46 experts in 1990. It also involves dispositions such as open-mindedness and inquisitiveness. (Facione, 1990)

Inquiry · Inquiry in education and learning
See also: Metacognition, Argument mapping, Logical fallacy
Read more: Critical Thinking: Definitions and Components · Teaching Critical Thinking: What Works

Cronbach's alpha

A coefficient of internal consistency for a multi-item scale, based on the number of items and the average inter-item covariance. Values closer to 1 indicate that items vary together. A high alpha does not prove that a scale measures a single construct, and alpha rises with the number of items. (Cronbach, 1951)

Statistics · Survey statistics
See also: Reliability, Likert scale, Factor analysis
Read more: Reliability Statistics: Cronbach's Alpha and Alternatives

Cross-validation

A method for estimating how well a model will predict new data by repeatedly fitting it on part of the data and testing it on the held-out remainder. In k-fold cross-validation the data are split into k parts and each part serves once as the test set. It is used to choose model complexity and avoid overfitting. (Hastie et al., 2009)

Statistics · Data science and machine learning
See also: Overfitting, Model selection criteria
Read more: Predictive Modelling: Training, Testing and Validation · Model Selection Criteria (AIC, BIC, Cross-Validation) and Their Epistemic Implications

Cumulative knowledge

Knowledge that builds on earlier work, so that each contribution extends, corrects or combines what came before. Robert Merton traced the aphorism "standing on the shoulders of giants", used by Newton, back to Bernard of Chartres in the twelfth century. (Merton, 1965)

The Generation of Knowledge · Knowledge production
See also: Peer review, Open science
Read more: Cumulative Knowledge: Standing on the Shoulders of Giants

Curiosity

The desire to know or learn. George Loewenstein's information-gap theory proposes that curiosity arises when attention is drawn to a gap between what one knows and what one wants to know, and that it rises as the gap narrows, which explains why a little knowledge often makes people more curious. (Loewenstein, 1994)

Inquiry · The art of the question
See also: Information-gap theory, Inquiry
Read more: Curiosity: The Psychology of Wanting to Know · The Information-Gap Theory of Curiosity

Customer effort score also: CES

A metric based on asking customers how much effort they had to make to get an issue resolved or a task done. Its proponents found that reducing customer effort predicted loyalty better than delighting customers. Like other single-question metrics, it is best interpreted alongside open-ended feedback. (Dixon et al., 2010)

Insights · Customer and market insight
See also: Net Promoter Score, Customer experience, Voice of the customer
Read more: Customer Effort Score and the Measurement of Service Friction · Customer Experience Insights: Measuring Moments That Matter

Customer experience also: CX

A customer's cognitive, emotional, sensory, social and behavioural responses to a company across every touchpoint of the customer journey, before, during and after purchase. Customer experience research maps these touchpoints and identifies the moments that most affect satisfaction and loyalty. (Lemon & Verhoef, 2016)

Insights · Customer and market insight
See also: Customer journey, Voice of the customer, Peak–end rule
Read more: Customer Experience Insights: Measuring Moments That Matter

Customer journey

The sequence of stages and touchpoints a customer passes through in becoming aware of, considering, buying, using and recommending or leaving a product or brand. Journey maps combine research evidence on each step to show where customers struggle or drop out. (Lemon & Verhoef, 2016)

Insights · Customer and market insight
See also: Customer experience
Read more: Purchase Decision Journeys and Touchpoint Insight · Customer Journey Research

Customer lifetime value also: CLV

The present value of all future profits expected from a customer relationship, usually modelled from retention probability, purchase frequency, margin and a discount rate. It shifts attention from single transactions to the long-term worth of customers and guides how much to spend acquiring and retaining them. (Gupta et al., 2006)

Insights · Customer and market insight
See also: Churn
Read more: Customer Lifetime Value and Insight-Led Retention

D

Dashboard

A visual display of the most important information needed to achieve one or more objectives, arranged on a single screen so it can be monitored at a glance. Good dashboards show a small number of decision-relevant indicators with context, such as targets and trends, rather than decoration. (Tufte, 1983; Chen et al., 2012)

Intelligence · Business intelligence
See also: Key performance indicator, Business intelligence
Read more: Dashboards That Inform Decisions Rather Than Decorate Them

Data

Recorded observations, measurements or symbols, such as survey responses or transaction records, that have not yet been organised or interpreted for a purpose. Data are never fully raw: what is recorded depends on choices about what to measure and how. (Kitchin, 2014; Zins, 2007)

The Generation of Knowledge · Knowledge production
See also: DIKW pyramid, Information
Read more: The DIKW Pyramid: Data, Information, Knowledge, Wisdom · Primary Data as the Raw Material of Knowledge

Deception also: denial and deception

Deliberate efforts by an adversary to mislead an analyst or decision-maker by concealing the truth (denial) or presenting falsehoods (deception), so that the target acts against its own interests. Analysts counter it by checking source access, looking for anomalies and testing whether evidence could have been planted. (Heuer, 1999)

Intelligence · Analysis and tradecraft
See also: Disinformation, Counterintelligence, Source evaluation
Read more: Deception and Disinformation: Counter-Narratives in Knowledge Generation

Decision framing

Defining the decision to be made, its scope, objectives, alternatives and constraints before seeking answers. Research on framing effects shows that how a choice is described changes what people choose, so getting the frame right is part of good inquiry. (Tversky & Kahneman, 1981)

Inquiry · Inquiry in organisations
See also: Framing effect
Read more: Decision Framing: Getting the Question Right Before Answering It

Decision-making under uncertainty

Choosing between options when outcomes are not known. When probabilities can be estimated (risk), expected value and expected utility guide choice; when they cannot (Knightian uncertainty), approaches such as scenario planning, robust strategies and gathering more information become important. (Knight, 1921; Tversky & Kahneman, 1981)

The Generation of Knowledge · Knowledge and decisions
See also: Value of information, Knightian uncertainty, Scenario planning
Read more: Decision-Making Under Uncertainty

Deduction also: deductive reasoning

Reasoning in which the conclusion follows necessarily from the premises: if the premises are true and the argument is valid, the conclusion must be true. Deduction preserves truth but does not add information beyond what the premises already contain. (Walton, 2008)

Inquiry · Logic and reasoning
See also: Induction, Abduction, Logical fallacy
Read more: Foundations of Logical Reasoning · Induction, Deduction and Abduction in Research Reasoning

Deliberative inquiry also: citizens' assembly

Inquiry in which a group of citizens, often randomly selected, learns about an issue from balanced information and experts, discusses it and reaches considered judgements. Deliberative polls and citizens' assemblies show that informed opinion can differ markedly from initial opinion. (Fishkin, 2009)

Inquiry · Public and formal inquiry
See also: Citizen science
Read more: Citizen Assemblies and Deliberative Inquiry

Descriptive, predictive and prescriptive analytics

A common classification of analytics by the question asked: descriptive (what happened), diagnostic (why it happened), predictive (what is likely to happen) and prescriptive (what should be done). Each stage requires more modelling and stronger assumptions than the one before. (Chen et al., 2012)

Intelligence · Business intelligence
See also: Business intelligence
Read more: Descriptive, Diagnostic, Predictive and Prescriptive Analytics

Design effect also: deff

The ratio of the variance of an estimate under a complex sample design to the variance it would have under a simple random sample of the same size. Clustering and unequal weights usually raise it above 1; effective stratification can lower it. Dividing the actual sample size by the design effect gives the effective sample size, the number of simple random cases with the same precision. (Kish, 1965; Kish, 1992)

Surveys · Sampling
See also: Effective sample size, Stratified sampling, Cluster sampling, Weighting
Read more: Design Effects and Effective Sample Size in Complex Surveys · Variance Estimation in Complex Survey Samples

Design thinking

A human-centred approach to innovation that uses empathy with users, problem framing, idea generation, prototyping and testing in iterative cycles. Its inquiry stance treats early solutions as questions to be tested with users rather than answers to be defended. (Brown, 2008)

Inquiry · Inquiry in organisations
See also: Lean startup, Jobs to be done
Read more: Design Thinking as a Process of Inquiry

Difference-in-differences also: DiD

A method that estimates a causal effect by comparing the change in an outcome over time in a group exposed to a policy or event with the change in an unexposed group. It removes stable differences between the groups and common trends, under the key assumption that the groups would have followed parallel trends without the intervention. (Card & Krueger, 1994; Angrist & Pischke, 2009)

Statistics · Causal inference
See also: Quasi-experiment, Synthetic control method, Counterfactual
Read more: Difference-in-Differences Analysis

Differential privacy

A mathematical definition of privacy for statistical releases: an algorithm is differentially private if its output distribution changes very little whether or not any one person's data are included. It is usually achieved by adding calibrated random noise, and it gives a measurable privacy budget, epsilon. The US Census Bureau used it for 2020 census products. (Dwork et al., 2006)

Statistics · Communication and misuse
Read more: Data Privacy in Statistics: Disclosure Control and Differential Privacy

Diffusion of innovations

Everett Rogers's theory of how new ideas and products spread through a social system over time, through adopter groups described as innovators, early adopters, early majority, late majority and laggards. Adoption depends on perceived advantage, compatibility, complexity, trialability and observability. (Rogers, 2003)

Insights · Tools and the future
Read more: From Insight to Innovation: Linking Understanding to New Products · Trend Insight: Separating Lasting Shifts From Fads

Digital forensics

The identification, preservation, collection, examination, analysis and presentation of digital evidence from computers, phones, networks and online services, using methods that maintain integrity and can withstand legal scrutiny. Work is done on verified copies so the original is not altered. (Casey, 2011)

Investigation · Digital investigation
See also: Chain of custody, Metadata
Read more: Digital Forensics: Principles and Process · Mobile Device Forensics and App Data Extraction

DIKW pyramid also: knowledge hierarchy

A model that arranges data, information, knowledge and wisdom as a hierarchy in which each level is derived from the one below. Russell Ackoff's 1989 article made it widely known. Critics argue that the levels are poorly defined and that knowledge is needed to produce data in the first place, so the relationships run in both directions. (Ackoff, 1989; Rowley, 2007; Frické, 2009)

The Generation of Knowledge · Knowledge production
See also: Data, Information, Knowledge
Read more: The DIKW Pyramid: Data, Information, Knowledge, Wisdom · How New Knowledge Is Created: From Data to Wisdom

Directed acyclic graph also: DAG

A diagram of variables (nodes) connected by arrows representing direct causal effects, with no path that returns to its starting point. DAGs make causal assumptions explicit and show which variables must be adjusted for to estimate an effect (the back-door criterion) and which must not, such as colliders and mediators. (Pearl, 2009; Pearl & Mackenzie, 2018)

Statistics · Causal inference
See also: Confounding, Collider bias
Read more: Directed Acyclic Graphs for Causal Reasoning

Disinformation

False information deliberately created and spread to deceive or cause harm. It is distinguished from misinformation, false information shared without intent to deceive, and malinformation, genuine information shared to cause harm. False news has been found to spread farther and faster online than true news. (Wardle & Derakhshan, 2017; Vosoughi et al., 2018)

Intelligence · Analysis and tradecraft
See also: Deception, Source evaluation
Read more: Deception and Disinformation: Counter-Narratives in Knowledge Generation · Investigating Disinformation Campaigns

Double-loop learning

Chris Argyris and Donald Schön's term for learning that questions and changes the underlying goals, norms and assumptions behind actions, as opposed to single-loop learning, which corrects actions within existing assumptions. A thermostat that adjusts heating is single-loop; questioning the temperature setting is double-loop. (Argyris & Schön, 1978)

Inquiry · Inquiry in organisations
See also: Psychological safety
Read more: Double-Loop Learning and Questioning Assumptions

Due diligence

A systematic investigation of a company, person or transaction before a decision such as an acquisition, investment, partnership or hire. It covers financial, legal, operational and reputational matters and aims to confirm what is claimed and uncover what is not disclosed. (Financial Action Task Force, 2012)

Investigation · Business and corporate investigation
See also: Anti-money laundering
Read more: Due Diligence Investigations in Mergers and Acquisitions · Reputation Due Diligence: Investigating What the Public Record Shows · Investigating Supplier and Third-Party Risk

E

Ecological fallacy

Inferring relationships about individuals from data on groups. William Robinson showed in 1950 that the correlation between literacy and foreign birth across US states had the opposite sign to the correlation among individuals. Group-level correlations can be strong even when individual-level ones are weak or reversed. (Robinson, 1950)

Statistics · Communication and misuse
See also: Simpson's paradox, Spurious correlation
Read more: Simpson's Paradox: When Aggregated Data Mislead · Correlation Is Not Causation: Spurious Relationships in Data

Effect size

A quantitative measure of the magnitude of a difference or relationship, such as a mean difference, Cohen's d, an odds ratio or a correlation. Unlike a p-value it does not depend on sample size, so it indicates practical importance. Jacob Cohen's conventions of small, medium and large effects are widely used but were meant only as a fallback. (Cohen, 1988)

Statistics · Inference
See also: Statistical power, P-value, Meta-analysis
Read more: Effect Sizes: Reporting What Matters Beyond Significance · Statistical Significance Versus Practical Significance

Effective sample size

The size of a simple random sample that would give the same precision as the actual sample, once clustering and weighting are taken into account. For weighting alone, Kish's approximation is the square of the sum of weights divided by the sum of squared weights. Reporting it shows how much precision a design or a weighting scheme has cost. (Kish, 1965; Kish, 1992)

Surveys · Sampling
See also: Design effect, Weighting
Read more: Design Effects and Effective Sample Size in Complex Surveys · Weighted Means and Weighted Percentages

Epidemiological investigation

Investigation of the occurrence, distribution and causes of disease in a population, including outbreak investigations that identify a source and route of transmission. John Snow's 1854 study of cholera around the Broad Street pump in London, which linked cases to one water source, is a founding example. (Snow, 1855)

Investigation · Scientific and technical investigation
See also: Root cause analysis, Confounding
Read more: Epidemiological Investigation: Tracing the Source of Outbreaks · John Snow and the Broad Street Pump: The First Modern Investigation of Disease

Epistemic injustice

Miranda Fricker's term for wrongs done to people in their capacity as knowers. Testimonial injustice occurs when prejudice leads a hearer to give a speaker less credibility; hermeneutical injustice occurs when a group lacks shared concepts to make sense of its experience. (Fricker, 2007)

The Generation of Knowledge · Knowledge and decisions
See also: Feminist inquiry, Social epistemology
Read more: Knowledge, Power and Who Gets to Know

Ethnography

The study of a group's culture and everyday practices through extended participant observation, interviews and documents, aiming to understand the world from members' perspectives. Clifford Geertz described its goal as "thick description": interpreting behaviour within the web of meanings that gives it significance. (Geertz, 1973)

Research · Qualitative research
See also: Netnography
Read more: Ethnography and Participant Observation in Commercial Research · Ethnographic Inquiry and Understanding From the Inside

Evidence

Information that tends to make a claim more or less probable. Investigators distinguish direct from circumstantial evidence, and testimonial, documentary, physical and digital evidence, and they assess each item for relevance, reliability and weight. A lead is a direction for inquiry; it becomes evidence only once verified. (National Research Council, 2009)

Investigation · Foundations
See also: Admissibility, Corroboration, Chain of custody
Read more: Evidence: Types, Weight and Admissibility · Evidence, Leads, and Claims: Keeping Their Roles Distinct

Evidence-based management

Making managerial decisions by combining the best available research evidence with organisational data, professional expertise and stakeholder values, and evaluating the results. Denise Rousseau's 2006 presidential address to the Academy of Management helped establish the idea. (Rousseau, 2006)

The Generation of Knowledge · Knowledge and decisions
See also: Systematic review, Actionable insight
Read more: Evidence-Based Management

Expert witness

A person whose specialised knowledge allows them to give opinion evidence to a court on matters beyond ordinary experience, such as forensic accounting or DNA analysis. Their duty is to the court rather than to the party that retains them, and their methods may be tested for reliability. (Daubert v. Merrell Dow Pharmaceuticals, Inc., 1993)

Investigation · Law, ethics and regulation
See also: Admissibility, Forensic science
Read more: The Role of Expert Witnesses in Translating Investigative Knowledge for Audiences

Explicit knowledge

Knowledge that can be articulated, codified and shared in words, numbers, documents, databases or procedures. It is easier to store and transfer than tacit knowledge, but it depends on tacit understanding to be interpreted and used. (Nonaka, 1994)

The Generation of Knowledge · Epistemology
See also: Tacit knowledge, SECI model
Read more: Tacit and Explicit Knowledge: Polanyi's Distinction

External validity

The extent to which a causal relationship found in a study holds across other people, settings, treatments and outcomes. Laboratory experiments can be strong on internal validity but weak on external validity; field experiments and replication across contexts strengthen it. (Shadish et al., 2002)

Research · Quantitative and mixed methods
See also: Internal validity, Field experiment
Read more: Internal and External Validity in Research Design · Field Experiments Versus Laboratory Experiments

Eyewitness testimony

Accounts of events given by people who saw them. Research shows that memory is reconstructive and can be altered by leading questions, post-event information and stress; for example, the verb used in a question changed witnesses' estimates of car speed. Mistaken eyewitness identification is a leading factor in wrongful convictions. (Loftus & Palmer, 1974; Wells & Olson, 2003)

Investigation · Foundations
See also: Cognitive interview, Investigative interviewing
Read more: Eyewitness Testimony and the Fallibility of Memory

F

Fact-checking

Verifying the factual accuracy of claims, either before publication within a news organisation or afterwards through public assessments of statements by politicians and others. Political fact-checking organisations grew rapidly from the 2000s and developed standard methods for sourcing and rating claims. (Graves, 2016; Kovach & Rosenstiel, 2014)

Investigation · Journalism and public investigation
See also: Corroboration, Disinformation
Read more: Fact-Checking: Methods and Effectiveness

Factor analysis

A family of methods that explain the correlations among many observed variables by a smaller number of unobserved (latent) factors. Exploratory factor analysis looks for structure; confirmatory factor analysis tests a structure specified in advance. Charles Spearman introduced the approach in 1904 in his work on general intelligence. (Spearman, 1904)

Statistics · Tests and models
See also: Principal component analysis, Measurement invariance, Cronbach's alpha
Read more: Factor Analysis: Exploratory and Confirmatory

FAIR data

Data that are Findable, Accessible, Interoperable and Reusable, according to guiding principles published in 2016. The principles emphasise rich metadata, persistent identifiers and clear licences so that both people and machines can find and reuse data. (Wilkinson et al., 2016)

The Generation of Knowledge · Trust, quality and the future
See also: Open science, Provenance
Read more: Preserving Knowledge for the Future: Archives and Digital Preservation · Research Data Management Plans and Long-Term Knowledge Preservation

Fake review

An online review written to mislead, for example by a business praising itself or attacking competitors. A study of Yelp found that about 16% of restaurant reviews were filtered as suspicious, and that businesses with weak reputations or more competition were more likely to commit review fraud. (Luca & Zervas, 2016)

Investigation · Market and consumer investigation
See also: Corroboration, Disinformation
Read more: Investigating Online Reviews and Fake Feedback

Fallibilism

The view that any belief, however well supported, could turn out to be mistaken and should remain open to revision in light of new evidence. Fallibilism does not imply scepticism: it holds that we can have knowledge while admitting that our methods are imperfect. (Peirce, 1877; Popper, 1959)

Inquiry · Philosophy of inquiry
See also: Falsifiability, Intellectual humility
Read more: Fallibilism: Holding Knowledge Open to Revision

False confession

An admission of guilt for a crime the person did not commit. Research distinguishes voluntary, compliant (given to escape pressure) and internalised (the person comes to believe they did it) false confessions, and links them to lengthy, accusatory interrogation, vulnerable suspects and presentation of false evidence. (Kassin & Gudjonsson, 2004)

Investigation · Foundations
See also: Investigative interviewing, PEACE model
Read more: Cognitive Interviewing versus Traditional Interrogation: Knowledge Yield Comparisons

False discovery rate also: FDR

The expected proportion of rejected null hypotheses that are actually true, that is, the share of "discoveries" that are false. The Benjamini–Hochberg procedure controls it when many tests are performed, offering more power than methods that control the chance of any false positive. (Benjamini & Hochberg, 1995)

Statistics · Inference
See also: Multiple comparisons, Type I error
Read more: Multiple Comparisons and the Problem of False Discoveries

Falsifiability

Karl Popper's criterion for distinguishing scientific from non-scientific claims: a theory is scientific only if it forbids some possible observations, so that it could in principle be shown false by evidence. On this view science advances by bold conjectures and attempts at refutation rather than by accumulating confirmations. (Popper, 1959)

Research · Philosophy and paradigms
See also: Hypothesis, Paradigm, Induction
Read more: Karl Popper and Falsifiability as a Criterion of Science · The Scientific Method: Hypothesis, Test and Revision

Feminist inquiry

Approaches to research that examine how gender and power shape what counts as knowledge, who is treated as a knower and which questions are asked. Standpoint theory, developed by Sandra Harding and others, argues that starting from the lives of marginalised groups can produce more objective knowledge. (Harding, 1986)

Inquiry · Philosophy of inquiry
See also: Epistemic injustice, Reflexivity
Read more: Feminist Inquiry: Re-evaluating Traditional Modes of Knowledge Generation

Field experiment

A randomised experiment conducted in a natural setting, such as a workplace, market or website, where participants often do not know they are in a study. Field experiments trade some control for greater realism and therefore stronger external validity than laboratory experiments. (Shadish et al., 2002)

Research · Research design
See also: Randomised controlled trial, A/B test, External validity
Read more: Field Experiments Versus Laboratory Experiments

File-drawer problem

The tendency for studies with null or non-significant results to remain unpublished, "in the file drawer", so that the published literature overstates the size and reliability of effects. Robert Rosenthal named the problem in 1979 and proposed estimating how many unpublished null studies would be needed to overturn a finding. (Rosenthal, 1979; Franco et al., 2014)

Research · Research integrity
See also: Publication bias, Registered report, Meta-analysis
Read more: Publication Bias and the File-Drawer Problem

Filter bubble

Eli Pariser's term for the isolation that can result when personalisation algorithms show people mainly information that matches their past behaviour and views. Empirical studies suggest that its effects are smaller than often feared, with self-selection of sources playing a large role. (Pariser, 2011)

The Generation of Knowledge · Trust, quality and the future
See also: Social epistemology, Disinformation
Read more: Filter Bubbles and Fragmented Knowledge

Finding

A result of analysis, such as "42% of customers rate delivery as poor". A finding becomes an insight when it is interpreted: when it explains why the result occurs and what it means for a decision. The distinction matters because reports full of findings can leave decision-makers no clearer about what to do. (Ackoff, 1989; Rowley, 2007)

Insights · What an insight is
See also: Insight, Actionable insight, DIKW pyramid
Read more: Data, Findings and Insights: Why the Difference Matters

Fishbone diagram also: Ishikawa diagram

A cause-and-effect diagram, introduced by Kaoru Ishikawa, that arranges possible causes of a problem along branches of categories, such as people, methods, machines, materials, measurement and environment. It helps teams consider causes broadly before testing which ones matter.

Investigation · Scientific and technical investigation
See also: Root cause analysis, Five whys
Read more: Root Cause Analysis Methods: Fishbone, Fault Tree and Five Whys

Five forces also: Porter's five forces

Michael Porter's framework for analysing the competitive structure of an industry through five forces: rivalry among existing competitors, the threat of new entrants, the threat of substitutes, and the bargaining power of buyers and of suppliers. The combined strength of the forces shapes the industry's profit potential. (Porter, 1979)

Intelligence · Market and competitive intelligence
See also: Competitive intelligence, SWOT analysis
Read more: Porter's Five Forces as an Intelligence Framework

Five whys

A root-cause technique from the Toyota Production System in which one asks "why?" repeatedly, typically about five times, to move from a symptom to its underlying cause. Its simplicity is its strength and its weakness: it can stop at a single chain of causes when problems usually have several. (Ohno, 1988)

Inquiry · The art of the question
See also: Root cause analysis, Fishbone diagram
Read more: The Five Whys and Root-Cause Questioning · Root Cause Analysis Methods: Fishbone, Fault Tree and Five Whys

Fixation of belief

Peirce's 1877 account of four methods by which people settle opinions: tenacity (holding to what one already believes), authority (accepting what institutions declare), the a priori method (accepting what seems agreeable to reason) and the method of science, which alone lets beliefs be corrected by something outside the believer. (Peirce, 1877)

Inquiry · Philosophy of inquiry
See also: Inquiry, Fallibilism
Read more: Charles Sanders Peirce and the Fixation of Belief

Forensic accounting

The application of accounting, auditing and investigative skills to legal questions such as fraud, embezzlement, financial statement manipulation, damages and disputes. Forensic accountants trace funds, analyse records and present findings suitable for use in court. (Hogan et al., 2008; Association of Certified Fraud Examiners, 2024)

Investigation · Business and corporate investigation
See also: Fraud triangle, Anti-money laundering
Read more: Forensic Accounting and Financial Investigation

Forensic science

The application of scientific methods to questions of law, through disciplines such as DNA analysis, toxicology, fingerprint, firearm and digital examination. A 2009 US National Research Council report found that, apart from nuclear DNA analysis, many forensic disciplines lacked rigorous validation of their accuracy. (National Research Council, 2009)

Investigation · Scientific and technical investigation
See also: Expert witness, Confirmation bias in investigation
Read more: Forensic Science: Strengths, Limits and Reform

Foundationalism

The view that justified beliefs rest on a foundation of basic beliefs that do not need justification from other beliefs, such as beliefs about immediate experience. Coherentism, its main rival, holds that beliefs are justified by how well they fit together in a system. (BonJour, 1985)

The Generation of Knowledge · Epistemology
See also: Coherentism, Methodical doubt
Read more: Foundationalism, Coherentism and the Structure of Knowledge

Framing effect

The finding that logically equivalent descriptions of a choice lead to different decisions, for example describing a treatment as saving 200 of 600 lives rather than letting 400 die. Framing shows that answers depend on how a question is asked, which matters for surveys, messaging and decisions. (Tversky & Kahneman, 1981)

Insights · Generating insight
See also: Behavioural economics
Read more: Heuristics and Biases in Consumer Decisions · Decision Framing: Getting the Question Right Before Answering It

Fraud diamond

An extension of the fraud triangle by Wolfe and Hermanson that adds capability: the personal traits and position needed to recognise and exploit an opportunity to commit fraud. (Wolfe & Hermanson, 2004)

Investigation · Business and corporate investigation
See also: Fraud triangle
Read more: The Fraud Triangle: Pressure, Opportunity and Rationalisation

Fraud triangle

A model, derived from Donald Cressey's 1953 study of embezzlers, holding that occupational fraud requires three conditions: pressure (a financial or other need), opportunity (weak controls) and rationalisation (a justification). It underlies auditing standards, although critics argue it neglects organisational and collective causes of fraud. (Cressey, 1953; Dorminey et al., 2012)

Investigation · Business and corporate investigation
See also: Fraud diamond, Forensic accounting
Read more: The Fraud Triangle: Pressure, Opportunity and Rationalisation · Fraud Investigation: Detection, Evidence and Recovery

Frequentist inference

An approach to statistics in which probability means long-run relative frequency, parameters are fixed but unknown, and methods are judged by how they would perform over repeated samples. Confidence intervals, p-values and hypothesis tests are frequentist tools. (Neyman, 1937)

Statistics · Foundations
See also: Bayesian inference, Confidence interval, P-value
Read more: Frequentist and Bayesian Statistics: Two Philosophies of Probability

G

Generalised linear model also: GLM

An extension of linear regression, introduced by Nelder and Wedderburn in 1972, in which the outcome can follow any distribution from the exponential family (normal, binomial, Poisson and others) and is related to the predictors through a link function. Logistic and Poisson regression are special cases. (Nelder & Wedderburn, 1972)

Statistics · Tests and models
See also: Logistic regression, Regression
Read more: Generalized Linear Models and Their Extensions for Diverse Outcome Types · Logistic Regression for Binary Outcomes

Gettier problem

The challenge, from Edmund Gettier's three-page 1963 paper, that a justified true belief can fail to be knowledge when it is true only by luck, for example a belief inferred from a false premise that happens to be correct. Attempts to add a fourth condition have generated a large literature. (Gettier, 1963)

The Generation of Knowledge · Epistemology
See also: Justified true belief
Read more: The Gettier Problem and the Definition of Knowledge

Grounded theory

A qualitative methodology, introduced by Barney Glaser and Anselm Strauss in 1967, for building theory from data rather than testing existing theory. It uses constant comparison between cases, open and focused coding, memo writing and theoretical sampling, continuing until new data no longer add to the emerging categories. (Glaser & Strauss, 1967)

Research · Qualitative research
See also: Thematic analysis, Saturation, Coding
Read more: Grounded Theory: Building Theory From Data

Groupthink

Irving Janis's term for a mode of thinking in cohesive groups where the desire for agreement overrides realistic appraisal of alternatives. Symptoms include an illusion of invulnerability, self-censorship, pressure on dissenters and stereotyped views of outsiders. Devil's advocacy, red teams and independent reviews are common remedies. (Janis, 1972)

Intelligence · Analysis and tradecraft
See also: Red teaming, Intelligence failure
Read more: Groupthink and Organisational Barriers to Good Intelligence

H

Hallucination

Output from an AI language model that is fluent and confident but not supported by its sources or by fact, such as an invented citation. Because hallucinations are hard to distinguish from accurate output, AI-assisted research and intelligence work requires verification of every factual claim against primary sources. (Ji et al., 2023; Bender et al., 2021)

Intelligence · Artificial and collective intelligence
See also: Source evaluation, Machine-generated knowledge
Read more: Risks of Automated Intelligence: Hallucination, Bias and Overconfidence · Large Language Models as Research and Intelligence Assistants

HARKing

Hypothesising After the Results are Known: presenting a hypothesis formed after seeing the data as if it had been stated in advance. It makes exploratory findings look confirmatory and inflates false-positive rates. Pre-registration makes it detectable. (Kerr, 1998)

Research · Research integrity
See also: P-hacking, Pre-registration, Questionable research practices
Read more: P-Hacking, HARKing and Questionable Research Practices

Hermeneutics also: hermeneutic inquiry

The theory and practice of interpretation, originally of texts and later of human action and experience. Hans-Georg Gadamer argued that understanding always starts from prejudgements and develops through a "fusion of horizons" between interpreter and text, in a cycle between the parts and the whole. (Gadamer, 1960/2004)

Inquiry · Philosophy of inquiry
See also: Interpretivism
Read more: Hermeneutic Inquiry: Generating Knowledge through Textual Interpretation

Heuristic inquiry

A qualitative research approach developed by Clark Moustakas in which the researcher investigates an experience they have lived, moving through phases of initial engagement, immersion, incubation, illumination, explication and creative synthesis. (Moustakas, 1990)

Inquiry · Philosophy of inquiry
See also: Reflexivity, Phenomenology
Read more: Heuristic Inquiry: Internal Search and the Personal Generation of Knowledge

Heuristics and biases

A research programme begun by Amos Tversky and Daniel Kahneman showing that people judge probability and value with simple rules of thumb, such as representativeness, availability and anchoring, which are often useful but produce systematic errors. It is a major source of insight into consumer and organisational decisions. (Tversky & Kahneman, 1974; Kahneman, 2011)

Insights · Generating insight
See also: Behavioural economics, Confirmation bias, Base rate neglect
Read more: Heuristics and Biases in Consumer Decisions

Hypothesis

A testable statement about the relationship between variables, derived from theory or prior observation, that research is designed to support or refute. A good hypothesis specifies the expected direction of an effect and could be shown false by the evidence. (Popper, 1959)

Research · Research design
See also: Falsifiability, Operationalisation, Pre-registration
Read more: From Research Question to Hypothesis: Operationalising Concepts · The Scientific Method: Hypothesis, Test and Revision

I

Implicit Association Test also: IAT

A computer-based task that measures the strength of automatic associations between concepts by comparing how quickly people sort words or images when categories are paired in different ways. It is widely used to study implicit attitudes, although its reliability for individuals and its ability to predict behaviour are debated. (Greenwald et al., 1998)

Insights · Generating insight
See also: Say–do gap
Read more: Implicit Measurement and Unconscious Attitudes

Imputation

Filling in missing values with plausible estimates so that complete-data methods can be used. Methods range from mean and hot-deck imputation to model-based and multiple imputation. Multiple imputation creates several completed datasets so that the extra uncertainty from missing data is reflected in standard errors. (Little & Rubin, 2019)

Surveys · Nonresponse, weighting and data quality
See also: Item nonresponse, Missing at random
Read more: Imputation of Missing Survey Data: Methods and Assumptions

Incubation

A period in which a problem is set aside after initial work, during which a solution may emerge. A meta-analysis found a reliable incubation effect, larger for creative problems with many possible solutions and when the break was filled with a light task. Graham Wallas made incubation the second of his four stages of creative thought. (Sio & Ormerod, 2009; Wallas, 1926)

Insights · What an insight is
See also: Aha moment, Wallas's four stages
Read more: Incubation and the Role of Rest in Generating Insight · Wallas's Four Stages of Creative Thought

Indicators and warnings also: I&W

A method of monitoring for observable signs that a specified development, such as an attack, a market shift or a supplier failure, is becoming more likely. Analysts define in advance which indicators would signal change and track them systematically, which reduces the risk of surprise. (Heuer, 1999; Ansoff, 1975)

Intelligence · Analysis and tradecraft
See also: Weak signal, Scenario planning
Read more: Indicators and Warnings: Detecting Change Early

Induction also: inductive reasoning

Reasoning from particular observations to general conclusions or predictions, such as inferring that all swans are white from many white swans. Inductive conclusions can be strong but never certain, and David Hume's problem of induction asks what justifies them at all. Popper's falsificationism was one response. (Popper, 1959; Walton, 2008)

Inquiry · Logic and reasoning
See also: Deduction, Abduction, Falsifiability
Read more: Foundations of Logical Reasoning · Induction, Deduction and Abduction in Research Reasoning

Information

Data organised and given meaning so that they are useful for a purpose. In Claude Shannon's technical sense, information is a reduction in uncertainty, measured in bits; in the semantic sense used in information science, it is meaningful, well-formed and, on some accounts, truthful data. (Shannon, 1948; Floridi, 2005)

The Generation of Knowledge · Knowledge production
See also: Data, Knowledge, DIKW pyramid
Read more: The DIKW Pyramid: Data, Information, Knowledge, Wisdom

Information asymmetry

A situation in which one party to a transaction knows more than the other. George Akerlof's "market for lemons" showed that when buyers cannot tell good used cars from bad ones, good cars may be driven from the market. Independent research, ratings and warranties reduce asymmetry. (Akerlof, 1970)

The Generation of Knowledge · Knowledge economy
See also: Value of information, Information good
Read more: Information Asymmetry and the Value of Knowledge in Markets

Information good

Anything that can be digitised, such as data, reports, software or music. Information goods are costly to produce but cheap to copy, and using them does not use them up, which leads to pricing by versioning and to the economic view of knowledge as partly a public good. (Shapiro & Varian, 1999; Arrow, 1962)

The Generation of Knowledge · Knowledge economy
See also: Knowledge economy, Information asymmetry
Read more: Information Goods: Pricing and Selling Knowledge · The Economics of Information: Knowledge as a Public Good · The Market for Data and Research Products

Information literacy

The set of abilities needed to find, evaluate, use and create information effectively and ethically, including understanding that authority is constructed and contextual and that research is a process of inquiry. The Association of College and Research Libraries' 2015 framework is a standard reference. (Association of College and Research Libraries, 2015)

The Generation of Knowledge · Trust, quality and the future
See also: Source evaluation, Critical thinking
Read more: Information Literacy as a Foundation for Knowledge

Information overload

The state in which the amount of information exceeds a person's or organisation's capacity to process it, so that decision quality falls rather than rises with more information. Causes include volume, poor quality and poor organisation of information; remedies include filtering, prioritisation and better presentation. (Eppler & Mengis, 2004)

Intelligence · Organisational intelligence
See also: Dashboard, Intelligence requirement
Read more: Information Overload and the Filtering Problem · Insight Fatigue and Information Overload in High-Volume Analytical Settings

Information-gap theory

Loewenstein's 1994 account of curiosity as a feeling of deprivation caused by a perceived gap in knowledge. It predicts that curiosity depends on knowing enough to notice the gap, and that it can be triggered deliberately by posing questions, presenting puzzles or revealing partial information. (Loewenstein, 1994)

Inquiry · The art of the question
See also: Curiosity
Read more: The Information-Gap Theory of Curiosity

Inquiry

The process of moving from doubt or an unsettled situation to a settled, warranted belief through questioning, investigation and reasoning. Charles Sanders Peirce described inquiry as the struggle to escape the irritation of doubt; John Dewey defined it as the controlled transformation of an indeterminate situation into a unified one. (Peirce, 1877; Dewey, 1938)

Inquiry · Philosophy of inquiry
See also: Fixation of belief, Fallibilism, Community of inquiry
Read more: The Nature of Inquiry: Doubt, Question and Belief · John Dewey's Logic of Inquiry

Inquiry-based learning

Teaching in which learners investigate questions, problems or scenarios rather than receiving information directly. Evidence suggests it works best when guided: minimally guided discovery tends to overload novices, while structured inquiry with scaffolding and feedback can be effective. (Kirschner et al., 2006; Hmelo-Silver, 2004)

Inquiry · Inquiry in education and learning
See also: Problem-based learning, Productive failure
Read more: Inquiry-Based Learning: Theory and Evidence · The Distinction Between Open and Guided Inquiry Designs

Insight

In psychology, a sudden restructuring of a problem that makes its solution apparent. In business, a new understanding of customers, markets or operations that explains behaviour and suggests what to do. Both senses share the idea of seeing something that was not evident in the data or the problem as first presented. (Kounios & Beeman, 2014; Duncker, 1945)

Insights · What an insight is
See also: Actionable insight, Aha moment, Restructuring, Finding
Read more: What Is an Insight? Definitions From Psychology and Business · Data, Findings and Insights: Why the Difference Matters · Insight Versus Analysis: Two Routes to Understanding

Instrumental variable also: IV

A variable that affects the treatment, is unrelated to unmeasured confounders, and affects the outcome only through the treatment. Under these assumptions it can be used to estimate a causal effect from observational data, typically the effect for "compliers" whose treatment is changed by the instrument. (Angrist et al., 1996)

Statistics · Causal inference
See also: Confounding, Natural experiment
Read more: Instrumental Variables

Intellectual capital

The intangible knowledge assets of an organisation, often divided into human capital (people's skills and knowledge), structural capital (processes, systems, data and intellectual property) and relational capital (relationships with customers and partners). Resource-based theory treats such valuable, rare and hard-to-imitate resources as sources of competitive advantage. (Barney, 1991)

The Generation of Knowledge · Knowledge economy
See also: Knowledge economy, Knowledge management
Read more: Intellectual Capital and Intangible Assets · Knowledge as a Competitive Advantage

Intellectual humility

Recognising that one's beliefs might be wrong and being attentive to the limits of one's evidence and knowledge. People higher in intellectual humility are more open to opposing views and better at judging the strength of arguments, according to research by Leary and colleagues. (Leary et al., 2017)

Inquiry · The art of the question
See also: Fallibilism, Critical thinking
Read more: Intellectual Humility and Open-Minded Inquiry

Intelligence

In the organisational sense, information that has been collected, evaluated and analysed to give a decision-maker an advantage, together with the organisation and activity that produce it. Sherman Kent described intelligence as knowledge, organisation and activity. It differs from raw information in being evaluated and tailored to a decision. (Kent, 1949)

Intelligence · Concepts and history
See also: Intelligence cycle, Business intelligence
Read more: What Is Intelligence? Information, Analysis and Decision Advantage · Data, Information, Intelligence and Knowledge: Drawing the Distinctions · Sherman Kent and the Founding of Intelligence Analysis as a Discipline

Intelligence cycle

The conventional model of the intelligence process as a cycle of direction (setting requirements), collection, processing, analysis and production, and dissemination, followed by feedback. Critics argue that real intelligence work is less sequential, with analysts and policymakers interacting throughout. (Hulnick, 2006; Kent, 1949)

Intelligence · Concepts and history
See also: Intelligence requirement, Intelligence
Read more: The Intelligence Cycle: Direction, Collection, Processing, Analysis and Dissemination · Criticisms of the Intelligence Cycle and Alternative Models

Intelligence failure

A failure to anticipate or correctly assess a significant development, such as the attack on Pearl Harbor. Studies emphasise that failures usually stem less from missing information than from ambiguous signals, mistaken assumptions, organisational barriers and decision-makers' reluctance to act on warning, which is why some failures are unavoidable. (Betts, 1978)

Intelligence · Analysis and tradecraft
See also: Groupthink, Key assumptions check, Weak signal
Read more: Intelligence Failures: Case Studies and Lessons

Intelligence requirement also: key intelligence question

A statement of the information a decision-maker needs, expressed as a question that collection and analysis can address. Key intelligence questions or topics prioritise effort, link the intelligence function to decisions and give a basis for measuring its value. (Hulnick, 2006)

Intelligence · Organisational intelligence
See also: Intelligence cycle
Read more: Intelligence Requirements: Asking Decision-Makers What They Need to Know

Internal validity

The extent to which a study supports the conclusion that a relationship between variables is causal, rather than the product of confounding, selection, history, maturation or other threats. Random assignment is the strongest protection; quasi-experimental designs address the threats with design features and analysis. (Shadish et al., 2002)

Research · Quantitative and mixed methods
See also: External validity, Confounding, Quasi-experiment
Read more: Internal and External Validity in Research Design

Interpretivism

A philosophical position holding that social reality is constructed through the meanings people give to their actions, so that research must understand those meanings from the participants' point of view rather than only measure behaviour. It underpins much qualitative research, including ethnography and phenomenology. (Geertz, 1973; Lincoln & Guba, 1985)

Research · Philosophy and paradigms
See also: Positivism, Ethnography, Hermeneutics
Read more: Positivism, Interpretivism and Pragmatism in Research Design

Inter-rater reliability

The degree to which different coders or raters give consistent judgements of the same material. Simple percentage agreement overstates it because some agreement occurs by chance; chance-corrected statistics such as Cohen's kappa or Krippendorff's alpha are preferred. (Cohen, 1960; Krippendorff, 2019)

Statistics · Survey statistics
See also: Cohen's kappa, Reliability, Content analysis
Read more: Inter-Rater Reliability: Cohen's Kappa and Related Measures

Investigative interviewing

Interviewing witnesses, victims and suspects with the aim of obtaining accurate and reliable information rather than a confession. It replaced accusatory interrogation in many countries after research linked coercive methods to false confessions. (Milne & Bull, 1999; Kassin & Gudjonsson, 2004)

Investigation · Foundations
See also: PEACE model, Cognitive interview, False confession
Read more: Investigative Interviewing: The PEACE Model · Cognitive Interviewing versus Traditional Interrogation: Knowledge Yield Comparisons

Item nonresponse

Missing answers to particular questions from people who otherwise took part in a survey, for example skipping an income question. It differs from unit nonresponse, where the whole interview is missing. It is usually handled by imputation or by analysis methods that make explicit assumptions about why the data are missing. (Little & Rubin, 2019; Groves et al., 2009)

Surveys · Nonresponse, weighting and data quality
See also: Unit nonresponse, Imputation, Missing at random
Read more: Unit Nonresponse Versus Item Nonresponse: Causes and Remedies · Imputation of Missing Survey Data: Methods and Assumptions

J

Jobs to be done also: JTBD

A framework that understands demand by asking what progress a customer is trying to make in a particular circumstance, the "job" for which they "hire" a product. It shifts attention from customer attributes and product features to the functional, social and emotional dimensions of the customer's situation. (Christensen et al., 2016)

Insights · Generating insight
See also: Persona
Read more: Jobs to Be Done: Understanding What Customers Hire Products For

Justified true belief also: JTB

The traditional analysis, traced to Plato's Theaetetus, that a person knows a proposition if it is true, they believe it, and they are justified in believing it. Edmund Gettier's 1963 paper gave cases where all three conditions hold by luck, and most philosophers now accept that something more is needed. (Gettier, 1963)

The Generation of Knowledge · Epistemology
See also: Gettier problem, Knowledge
Read more: What Is Knowledge? Justified True Belief and Its Challenges · The Gettier Problem and the Definition of Knowledge

K

Key assumptions check

A structured technique in which analysts list the assumptions underlying a judgement, ask how confident they are in each and what would make it untrue, and identify which ones the conclusion depends on. It brings hidden premises into the open before they cause a failure. (Heuer, 1999)

Intelligence · Analysis and tradecraft
See also: Analysis of competing hypotheses, Structured analytic techniques
Read more: Key Assumptions Checks

Key performance indicator also: KPI

A measure chosen to track progress towards an important objective, such as customer retention or cost per acquisition. Good KPIs are few, clearly defined, linked to strategy and balanced, as in the balanced scorecard's financial, customer, internal process and learning perspectives. Targets set on a measure can distort behaviour. (Kaplan & Norton, 1992)

Intelligence · Business intelligence
See also: Dashboard, Business intelligence
Read more: Key Performance Indicators: Selection, Design and Misuse

Knightian uncertainty also: radical uncertainty

Uncertainty that cannot be measured by known probabilities, as distinguished by Frank Knight in 1921 from risk, where the probabilities of outcomes can be estimated. The phrase "unknown unknowns" refers to the extreme case where the possible outcomes themselves are not known. (Knight, 1921)

Inquiry · Ethics and limits
See also: Decision-making under uncertainty, Scenario planning
Read more: Uncertainty, Ignorance and Unknown Unknowns · Decision-Making Under Uncertainty

Know-how also: know-that

Practical knowledge of how to do something, distinguished by Gilbert Ryle in 1949 from know-that, knowledge of facts. Ryle argued that intelligent action is not just the application of known facts, a claim still debated in epistemology. (Ryle, 1949)

The Generation of Knowledge · Epistemology
See also: Tacit knowledge
Read more: Know-How Versus Know-That

Knowledge

In the classical philosophical analysis, justified true belief; the Gettier problem shows that this definition is incomplete. In organisations, knowledge is understanding that lets people act effectively, combining information with experience, context and judgement, and it includes both explicit and tacit forms. (Gettier, 1963; Nonaka, 1994)

The Generation of Knowledge · Epistemology
See also: Justified true belief, Tacit knowledge, DIKW pyramid
Read more: What Is Knowledge? Justified True Belief and Its Challenges

Knowledge economy

An economy in which production and services depend mainly on knowledge-intensive activities, intellectual capabilities and rapid technical change rather than physical inputs. Fritz Machlup first measured the share of the US economy devoted to producing and distributing knowledge in 1962. (Powell & Snellman, 2004; Machlup, 1962)

The Generation of Knowledge · Knowledge economy
See also: Intellectual capital, Information good
Read more: The Knowledge Economy: Concepts and Measurement

Knowledge graph

A representation of knowledge as a graph of entities (such as people, places and concepts) connected by typed relationships, often with a schema or ontology defining the types. Knowledge graphs support search, data integration and reasoning, and are used by search engines and in enterprise data management. (Hogan et al., 2021)

The Generation of Knowledge · Trust, quality and the future
See also: Provenance
Read more: Knowledge Graphs and Machine-Readable Knowledge

Knowledge management also: KM

The practices and systems by which organisations create, capture, store, share and apply knowledge, ranging from document repositories and expert directories to communities of practice and lessons-learned processes. Research suggests that technology alone rarely succeeds without attention to culture and incentives. (Alavi & Leidner, 2001)

The Generation of Knowledge · Knowledge management
See also: SECI model, Organisational memory, Communities of practice
Read more: Knowledge Management: Concepts and Practice

Knowledge transfer

The process by which knowledge moves from one person, team or organisation to another and is put to use. Transfer is harder for tacit, complex and context-specific knowledge, and depends on the receiver's absorptive capacity. (Cohen & Levinthal, 1990; Nonaka, 1994)

The Generation of Knowledge · Knowledge management
See also: Absorptive capacity, Tacit knowledge
Read more: Knowledge Transfer Between Organisations

L

Lasso

Least absolute shrinkage and selection operator: a regression method that adds a penalty proportional to the sum of the absolute values of the coefficients. The penalty shrinks coefficients towards zero and sets some exactly to zero, so it performs variable selection and reduces overfitting when there are many predictors. (Tibshirani, 1996; Hastie et al., 2009)

Statistics · Data science and machine learning
See also: Overfitting, Regression
Read more: Penalized Regression (Lasso, Ridge, Elastic Net) for High-Dimensional Knowledge

Latent class analysis

A statistical model that assumes the population consists of a number of unobserved classes and estimates, from categorical survey answers, the probability of each answer in each class and each respondent's probability of class membership. It is a model-based alternative to cluster analysis for segmentation. (Hastie et al., 2009)

Statistics · Tests and models
See also: Cluster analysis, Segmentation
Read more: Latent Class Analysis

Law of large numbers

The theorem that the average of independent observations from the same distribution converges to the expected value as the number of observations grows. It explains why large random samples give stable estimates, and it does not imply that short runs "even out", the gambler's fallacy. (Jaynes, 2003)

Statistics · Foundations
See also: Central limit theorem, Sampling distribution
Read more: The Law of Large Numbers and Its Practical Meaning

Lean startup

An approach to building new businesses that treats a business plan as a set of hypotheses to be tested quickly with customers, using a build–measure–learn loop and a minimum viable product. Its aim is validated learning: evidence about what customers actually want before large investment. (Ries, 2011)

Inquiry · Inquiry in organisations
See also: Design thinking
Read more: Lean Startup and Validated Learning

Levels of measurement also: scales of measurement

Stanley Stevens's classification of variables as nominal (categories), ordinal (ordered categories), interval (equal differences, arbitrary zero) or ratio (equal differences, true zero). The level determines which statistics are meaningful, although applying the rules strictly, for example to rating scales, remains debated. (Stevens, 1946)

Statistics · Foundations
See also: Likert scale
Read more: Levels of Measurement: Nominal, Ordinal, Interval and Ratio

Leverage-saliency theory

A theory of survey participation proposing that each feature of a survey request (topic, sponsor, incentive, burden) has a leverage, the weight a given person attaches to it, and a saliency, how prominent it is made in the request. A person's decision to take part depends on the combination, which is why the same design can attract some groups and deter others and why topic interest can produce nonresponse bias. (Groves et al., 2000; Groves et al., 2004)

Surveys · Incentives and engagement
See also: Nonresponse bias
Read more: Leverage-Saliency Theory and the Decision to Participate in a Survey

Lie detection also: deception detection

Attempts to tell truthful from deceptive statements by behaviour, speech or physiology. Research shows that people, including professionals, detect lies only slightly better than chance from nonverbal cues, and that methods focusing on the content and verifiability of statements are more promising. (Vrij, 2008)

Investigation · Scientific and technical investigation
See also: Investigative interviewing
Read more: Psychological and Behavioral Analysis Contributions to Investigative Knowledge

Likert scale

A multi-item measure introduced by Rensis Likert in 1932 in which respondents indicate their degree of agreement with a set of statements, and the item scores are summed or averaged into a scale score. In everyday use the term is often applied to a single agree–disagree item, which is more precisely a Likert-type item. Whether such data can be treated as interval rather than ordinal is a long-running debate. (Likert, 1932)

Surveys · Questionnaire design
See also: Acquiescence bias, Levels of measurement, Cronbach's alpha
Read more: Likert Scales: Construction, Interpretation and Common Misuses · Analysing Likert Scale Data: Ordinal or Interval?

Logical fallacy

A pattern of reasoning that seems persuasive but fails to support its conclusion, either because of its form (formal fallacies such as affirming the consequent) or because of its content and context (informal fallacies such as ad hominem, straw man, false dilemma or appeal to authority). Recognising fallacies is a core skill of critical thinking. (Walton, 2008)

Inquiry · Logic and reasoning
See also: Critical thinking, Argument mapping
Read more: Common Logical Fallacies and How to Recognise Them

Logistic regression

A regression model for a binary outcome in which the log-odds of the outcome is a linear function of the predictors. Exponentiated coefficients are odds ratios. It is a generalised linear model with a binomial distribution and a logit link. (Nelder & Wedderburn, 1972)

Statistics · Tests and models
See also: Generalised linear model, Odds ratio
Read more: Logistic Regression for Binary Outcomes

M

Machine-generated knowledge

Claims, summaries or predictions produced by algorithms and AI models rather than directly by people. Language models can produce fluent text that is not grounded in fact, which raises questions about whether such systems know anything and requires human verification, transparency about sources and accountability. (Bender et al., 2021; Ji et al., 2023)

The Generation of Knowledge · Trust, quality and the future
See also: Hallucination, Provenance
Read more: Machine-Generated Knowledge: Can AI Know Anything? · Artificial Intelligence and the Future of Knowledge Production

Margin of error

The half-width of a confidence interval for a survey estimate, usually at 95% confidence. For a proportion from a simple random sample it is about 1.96 times the standard error. It reflects sampling error only: it says nothing about coverage, nonresponse or measurement error, and for nonprobability samples its meaning depends on model assumptions. (Lohr, 2021; Neyman, 1934)

Surveys · Sampling
See also: Confidence interval, Standard error, Total survey error
Read more: The Margin of Error in Survey Results

Market intelligence

Information and analysis about a market's size, growth, segments, customers, competitors, prices and trends, gathered from primary research, secondary sources and internal data to inform market entry, product and strategy decisions. (Chen et al., 2012)

Intelligence · Market and competitive intelligence
See also: Competitive intelligence
Read more: Market Intelligence: Understanding Size, Trends and Dynamics

Measurement error

The difference between a recorded answer and the true value of what was meant to be measured, arising from question wording, respondents' memory and motives, interviewers, mode or data processing. Random measurement error adds noise and weakens correlations; systematic error biases estimates. (Groves & Lyberg, 2010; Tourangeau et al., 2000)

Surveys · Foundations and history
See also: Total survey error, Social desirability bias, Mode effect
Read more: Measurement Error and Its Effects on Statistical Estimates · The Total Survey Error Framework: A Unified Model of Survey Quality

Measurement invariance

The property that a measure assesses the same construct in the same way across groups, modes or time points. It is usually tested in steps (configural, metric, scalar) with multi-group confirmatory factor analysis. Without at least scalar invariance, differences in group means may reflect differences in how the questions work rather than true differences. (King et al., 2004)

Surveys · Nonresponse, weighting and data quality
See also: Anchoring vignettes, Factor analysis, Mode effect
Read more: Measurement Invariance Across Groups in Survey Research

Member checking also: respondent validation

Returning data, interpretations or findings to participants to ask whether they recognise them as accurate accounts of their experience. Lincoln and Guba described it as the most crucial technique for establishing credibility, though participants' views may legitimately differ from the researcher's analysis. (Lincoln & Guba, 1985)

Research · Qualitative research
See also: Trustworthiness
Read more: The Use of Member Checking and Respondent Validation for Knowledge Confirmation

Meta-analysis

A statistical method for combining the results of multiple studies of the same question into a pooled estimate of effect size, weighting each study by its precision. Fixed-effect models assume one true effect; random-effects models allow the true effect to vary between studies. Heterogeneity statistics and checks for publication bias are standard parts of the method. (Borenstein et al., 2009)

Research · Literature and evidence
See also: Systematic review, Effect size, Publication bias
Read more: Meta-Analysis: Combining Results Across Studies · Meta-Regression Techniques for Explaining Heterogeneity in Aggregated Knowledge

Metacognition

Thinking about one's own thinking: knowledge of one's cognitive processes and the monitoring and control of them, such as noticing that one has not understood a passage. John Flavell introduced the concept in 1979. Good metacognition helps inquirers judge when they know enough and when to ask more. (Flavell, 1979)

Inquiry · Inquiry in education and learning
See also: Critical thinking, Intellectual humility
Read more: Metacognition: Thinking About One's Own Thinking

Metadata

Data that describe other data, such as a file's creation date, author, device or GPS location, or an email's routing headers. Metadata can establish when, where and by whom digital content was created, and it can also be altered, so it must be examined critically. (Casey, 2011)

Investigation · Digital investigation
See also: Digital forensics
Read more: Metadata as Evidence

Methodical doubt

René Descartes's method, set out in his 1641 Meditations, of withholding assent from any belief that could possibly be doubted in order to find a foundation that is certain. It led to his conclusion "I think, therefore I am" and became a model for systematic scepticism in inquiry. (Descartes, 1641/1996)

Inquiry · Philosophy of inquiry
See also: Fallibilism, Foundationalism
Read more: Descartes and Methodical Doubt

Missing at random also: MAR

In Rubin's classification of missing data, the condition that the probability a value is missing depends only on observed data, not on the missing value itself. Data are missing completely at random if missingness is unrelated to any data, and missing not at random if it depends on the unobserved value. Most imputation and weighting methods assume missing at random. (Little & Rubin, 2019)

Surveys · Nonresponse, weighting and data quality
See also: Imputation, Item nonresponse, Nonresponse bias
Read more: Imputation of Missing Survey Data: Methods and Assumptions · Unit Nonresponse Versus Item Nonresponse: Causes and Remedies

Mixed methods research

Research that deliberately combines qualitative and quantitative data and analysis within one study or programme, and integrates the two. Core designs are convergent (both collected together and compared), explanatory sequential (qualitative work explains quantitative results) and exploratory sequential (qualitative work informs a later quantitative phase). (Creswell & Plano Clark, 2018)

Research · Quantitative and mixed methods
See also: Triangulation, Thematic analysis
Read more: Mixed Methods Research: Convergent, Explanatory and Exploratory Designs

Mixed-mode survey

A survey that collects data through more than one mode, such as web and telephone, either by offering a choice, by following up nonrespondents in a second mode, or by using different modes for different parts of the interview. Mixing modes can improve coverage and response but can introduce mode effects that make answers from different modes hard to compare. (de Leeuw, 2005; Dillman et al., 2014)

Surveys · Nonresponse, weighting and data quality
See also: Mode effect, Coverage error
Read more: Mixed-Mode Survey Administration: Harmonizing Data for Cohesive Knowledge · The Evolution of Survey Modes: Face-to-Face, Telephone, Mail and Web

Mode 2 knowledge production

A form of knowledge production described by Michael Gibbons and colleagues in 1994 that is organised around problems in a context of application, is transdisciplinary, involves diverse organisations and is subject to wider social accountability. It contrasts with Mode 1, discipline-based research within universities. (Gibbons et al., 1994)

The Generation of Knowledge · Knowledge production
See also: Triple helix
Read more: Mode 1 and Mode 2 Knowledge Production

Mode effect

A difference in survey answers caused by the mode of data collection (face-to-face, telephone, mail, web, app) rather than by differences between respondents. Interviewer-administered modes tend to produce more socially desirable answers on sensitive topics; visual and aural presentation can change how response scales are used. (de Leeuw, 2005; Tourangeau & Yan, 2007)

Surveys · Nonresponse, weighting and data quality
See also: Mixed-mode survey, Social desirability bias
Read more: Mode Effects: Why the Same Question Gives Different Answers by Channel

Model selection criteria also: AIC, BIC

Measures for choosing among statistical models that balance fit against complexity. The Akaike information criterion (AIC) estimates out-of-sample prediction error; the Bayesian information criterion (BIC) penalises extra parameters more heavily as sample size grows. Lower values indicate a preferred model. (Akaike, 1974)

Statistics · Data science and machine learning
See also: Cross-validation, Overfitting
Read more: Model Selection Criteria (AIC, BIC, Cross-Validation) and Their Epistemic Implications

Monte Carlo simulation

Using repeated random sampling to approximate quantities that are hard to calculate exactly, such as the distribution of outcomes of a risky project or the behaviour of a statistical method. Metropolis and Ulam named and described the method in 1949, drawing on work at Los Alamos. (Metropolis & Ulam, 1949)

Statistics · Bayesian and computational
See also: Bootstrap
Read more: Monte Carlo Simulation for Risk and Uncertainty · Simulation Studies for Evaluating Statistical Method Performance

MRP also: multilevel regression and post-stratification

Multilevel regression and post-stratification: a method that fits a multilevel model predicting a survey outcome from demographic and geographic variables, then weights the predictions for each population cell by its known size from census data. It produces estimates for small groups and areas and can correct for unrepresentative samples. (Park et al., 2004)

Statistics · Bayesian and computational
See also: Multilevel model, Weighting, Small area estimation
Read more: Multilevel Regression and Post-Stratification (MRP)

Multilevel model also: hierarchical model

A regression model for data with a nested structure, such as employees within firms or repeated measures within people, that includes random effects at each level. It estimates variation within and between groups and "partially pools" estimates for small groups towards the overall mean. (Gelman & Hill, 2007)

Statistics · Tests and models
See also: MRP, Small area estimation
Read more: Multilevel and Hierarchical Models · Bayesian Hierarchical Models for Partial Pooling and Knowledge Stabilization

Multiple comparisons

The problem that when many hypothesis tests are performed, the chance that at least one gives a false positive grows well above the nominal significance level. Corrections such as Bonferroni (controlling the family-wise error rate) and Benjamini–Hochberg (controlling the false discovery rate) adjust for it. (Benjamini & Hochberg, 1995)

Statistics · Inference
See also: False discovery rate, P-hacking, Type I error
Read more: Multiple Comparisons and the Problem of False Discoveries

Mystery shopping

A research method in which trained observers pose as ordinary customers to assess service quality, compliance or pricing. Because staff do not know they are being observed, it records actual behaviour, but it raises ethical questions about deception and consent. (Wilson, 2001)

Investigation · Market and consumer investigation
See also: Say–do gap
Read more: Mystery Shopping as Investigative Research

N

Natural experiment

A situation in which an external event, policy or rule assigns people to different conditions in a way that is as good as random, allowing causal effects to be studied without the researcher controlling assignment. Lotteries, arbitrary boundaries and sudden policy changes are common sources. (Angrist & Pischke, 2009; Card & Krueger, 1994)

Research · Research design
See also: Quasi-experiment, Instrumental variable, Difference-in-differences
Read more: Natural Experiments in Business and Social Research

Net Promoter Score also: NPS

A customer loyalty metric based on one question asking how likely a customer is to recommend a company on a 0–10 scale. The score is the percentage of promoters (9–10) minus the percentage of detractors (0–6). It was popularised as a predictor of growth, but later research found it no better than other satisfaction and loyalty measures at predicting company revenue growth. (Reichheld, 2003; Keiningham et al., 2007)

Surveys · Types and applications
See also: Customer effort score, Voice of the customer
Read more: Net Promoter Score: Origins, Evidence and Criticisms

Netnography

Ethnographic research adapted to online communities and social media, introduced by Robert Kozinets for marketing research. It studies the shared meanings and practices of online groups through observation, participation and analysis of their communications, with attention to consent and privacy. (Kozinets, 2002)

Research · Qualitative research
See also: Ethnography
Read more: Netnography: Studying Online Communities · Online Qualitative Research: Communities, Forums and Asynchronous Groups

Neuromarketing

The use of neuroscience and physiological methods, such as brain imaging, EEG, eye tracking and skin conductance, to study consumer responses to products, brands and advertising. Reviews find that it can reveal processes that people cannot report, but that many commercial claims outrun the evidence. (Plassmann et al., 2012)

Insights · Generating insight
See also: Implicit Association Test
Read more: Neuromarketing: Physiological Data as a Source of Consumer Knowledge

Nonprobability sample

A sample in which units are selected by a process that does not give every member of the population a known, non-zero chance of selection, such as opt-in online panels, quota samples or convenience samples. Estimates from such samples rely on modelling assumptions, typically that weighting or matching variables account for how participants differ from the population. (Baker et al., 2013)

Surveys · Sampling
See also: Probability sample, Quota sampling, Weighting
Read more: Non-Probability Sampling: Convenience, Quota, Snowball and Purposive Methods · Opt-In Online Samples: Can They Produce Valid Estimates? · Hybrid Probability-Nonprobability Designs for Cost-Efficient Knowledge Generation

Nonresponse bias

The difference between a survey estimate and the value it would have had with full response, caused by respondents differing from nonrespondents on the variable being measured. It depends on the correlation between response propensity and the survey variable, not on the response rate alone, which is why low-response surveys can be accurate and high-response surveys can be biased. (Groves, 2006; Groves & Peytcheva, 2008)

Surveys · Nonresponse, weighting and data quality
See also: Response rate, Unit nonresponse, Leverage-saliency theory, Weighting
Read more: Nonresponse Bias: When Low Response Rates Matter and When They Do Not · The Decline in Survey Response Rates and What It Means for Accuracy

Normal distribution also: Gaussian distribution

The symmetric, bell-shaped continuous distribution defined by its mean and standard deviation, with about 68% of values within one standard deviation of the mean and about 95% within two. Its importance comes largely from the central limit theorem. (Lohr, 2021)

Statistics · Distributions
See also: Central limit theorem, Standard deviation
Read more: The Normal Distribution and Why It Appears Everywhere

Norms of science also: CUDOS

Robert Merton's 1942 description of the ethos of science: communalism (findings are shared), universalism (claims are judged by impersonal criteria), disinterestedness (science serves knowledge, not personal gain) and organised scepticism (claims are scrutinised before acceptance). (Merton, 1973)

The Generation of Knowledge · Knowledge production
See also: Peer review, Open science
Read more: Peer Review, Verification and Quality Control of Knowledge · The Royal Society and the Institutionalisation of Science

Nudge

Any aspect of choice architecture that changes behaviour in a predictable way without forbidding options or significantly changing economic incentives, such as making a desirable option the default. Thaler and Sunstein popularised the term in 2008. (Thaler & Sunstein, 2008)

Insights · Generating insight
See also: Behavioural economics
Read more: Nudge Theory and Choice Architecture

O

Occam's razor also: principle of parsimony

The principle of parsimony: among explanations that account for the evidence equally well, prefer the one that makes fewer assumptions. It is a guide for choosing between hypotheses, not a guarantee that the simpler one is true. Model selection criteria such as AIC formalise a similar trade-off. (Lipton, 2004)

Inquiry · Logic and reasoning
See also: Abduction, Model selection criteria
Read more: Occam's Razor and the Principle of Parsimony

Odds ratio

The ratio of the odds of an outcome in one group to the odds in another; a value of 1 means no association. It is the natural effect measure in logistic regression and case–control studies. When the outcome is common, an odds ratio exaggerates the corresponding risk ratio. (Nelder & Wedderburn, 1972)

Statistics · Tests and models
See also: Logistic regression, Effect size
Read more: Logistic Regression for Binary Outcomes

Open science

Practices that make research methods, data, materials, code and publications openly available so that others can check, reuse and build on them. It includes open access publishing, open data following the FAIR principles (findable, accessible, interoperable, reusable), pre-registration and shared analysis code. (Munafò et al., 2017; Wilkinson et al., 2016)

Research · Research integrity
See also: Pre-registration, Replication crisis, FAIR data
Read more: Open Science: Open Data, Open Materials and Open Access · Research Data Management Plans and Long-Term Knowledge Preservation

Open-source intelligence also: OSINT

Intelligence derived from publicly available information, such as media, public records, company filings, academic work, satellite imagery and social media. The growth of the internet has made it the largest source for many questions; its main challenges are volume, verification and deception. (Glassman & Kang, 2012)

Intelligence · Open-source intelligence
See also: Source evaluation, All-source intelligence
Read more: Open-Source Intelligence (OSINT): Methods and Sources · OSINT for Business: Public Records, Filings and Media · Verifying Open-Source Information

Operationalisation

The process of turning an abstract concept into specific, observable measures or procedures, for example defining "customer loyalty" as repeat purchase within twelve months. Good operationalisation is explicit, so others can replicate the study, and valid, so the measure captures the concept. (Shadish et al., 2002)

Research · Research design
See also: Construct validity, Hypothesis
Read more: From Research Question to Hypothesis: Operationalising Concepts

Organisational intelligence

An organisation's capacity to gather, interpret and use information about its environment to make decisions. Harold Wilensky's 1967 study showed how hierarchy, specialisation and secrecy distort the flow of intelligence in government and industry. (Wilensky, 1967; Huber, 1991)

Intelligence · Organisational intelligence
See also: Double-loop learning, Intelligence
Read more: Organisational Intelligence: How Companies Learn and Sense Their Environment

Organisational memory

Stored information from an organisation's history that can be brought to bear on present decisions, held in individuals, culture, procedures, structures and records. Knowledge is lost when it lives only in people who leave, which is why documentation and succession matter. (Walsh & Ungson, 1991)

The Generation of Knowledge · Knowledge management
See also: Knowledge management
Read more: Organisational Memory and Knowledge Loss

Overfitting

Fitting a model so closely to the particular sample that it captures noise as well as signal, so that it performs well on the training data but poorly on new data. It is the high-variance side of the bias–variance trade-off and is controlled by cross-validation, regularisation and simpler models. (Hastie et al., 2009)

Statistics · Data science and machine learning
See also: Lasso, Cross-validation
Read more: Overfitting and the Bias-Variance Trade-off · Predictive Modelling: Training, Testing and Validation

Overgeneralisation

Drawing broad conclusions from too few or unrepresentative cases. Experiments show that people generalise from a single vivid case even when told it is atypical, which is why anecdotes and small qualitative samples need to be checked against representative data before they guide decisions. (Hamill et al., 1980; Tversky & Kahneman, 1974)

Insights · Insight quality and pitfalls
See also: Nonprobability sample, Apophenia
Read more: Overgeneralisation From Small or Unrepresentative Samples · The Danger of Anecdote-Driven Insight

P

Paradata

Data about the process of collecting survey data, such as call records, timestamps, keystrokes, device type and interviewer observations. Paradata are used to monitor fieldwork, detect data quality problems, model response propensity and adjust for nonresponse. (Kreuter, 2013)

Surveys · Nonresponse, weighting and data quality
See also: Responsive design, Careless responding
Read more: Using Paradata to Understand the Survey Response Process

Paradigm

In Thomas Kuhn's account, the shared set of theories, methods, exemplary problems and standards that defines normal science in a field. Accumulating anomalies can lead to a crisis and a scientific revolution in which one paradigm replaces another. In social research the word is also used for broad positions such as positivism, interpretivism and pragmatism. (Kuhn, 1962)

Research · Philosophy and paradigms
See also: Falsifiability, Positivism, Interpretivism
Read more: Thomas Kuhn, Paradigm Shifts and the Structure of Scientific Revolutions · Positivism, Interpretivism and Pragmatism in Research Design

PEACE model

A framework for investigative interviewing adopted in England and Wales in the early 1990s: Planning and preparation, Engage and explain, Account clarification and challenge, Closure, and Evaluation. It emphasises open questions, fairness and information gathering rather than confession seeking. (Milne & Bull, 1999)

Investigation · Foundations
See also: Investigative interviewing, Cognitive interview
Read more: Investigative Interviewing: The PEACE Model

Peak–end rule

The finding that people judge a past experience largely by its most intense moment and its end, rather than by the sum of every moment. In one study, participants preferred to repeat a longer unpleasant experience that ended less painfully. It explains why service recoveries and endings matter disproportionately to customer satisfaction. (Kahneman et al., 1993)

Insights · Customer and market insight
See also: Customer experience
Read more: Customer Experience Insights: Measuring Moments That Matter

Peer review

The evaluation of research by independent experts before publication or funding. It is meant to catch errors and judge importance, but studies show it is slow, inconsistent between reviewers, poor at detecting fraud, and open to bias. Open review, registered reports and post-publication review are attempts to improve it. (Smith, 2006)

Research · Research integrity
See also: Registered report, Replication crisis
Read more: Peer Review: How It Works and How It Fails · Peer Review, Verification and Quality Control of Knowledge

Persona

A fictional but evidence-based profile of a typical user or customer segment, describing goals, behaviours, needs and frustrations. Alan Cooper popularised personas in software design. They are useful when built from research data and misleading when they encode assumptions or stereotypes. (Cooper, 1999)

Insights · Communicating insight
See also: Segmentation, Jobs to be done
Read more: Personas: Building Them From Evidence Rather Than Assumption

P-hacking

Analysing data in many ways (adding or dropping observations, trying different covariates, outcomes or subgroups) and reporting the analyses that reach statistical significance. Even without intent to deceive, such flexibility can make false-positive findings very likely. (Simmons et al., 2011; John et al., 2012)

Research · Research integrity
See also: HARKing, Pre-registration, P-value
Read more: P-Hacking, HARKing and Questionable Research Practices

Phenomenology

A philosophical tradition and qualitative research approach that studies the structures of lived experience as it appears to the person, setting aside prior theories about it. Phenomenological research describes what an experience is like and what it means to those who have it. (Moustakas, 1990)

Inquiry · Philosophy of inquiry
See also: Hermeneutics, Heuristic inquiry
Read more: Phenomenology and the Study of Lived Experience

Positivism

A philosophical position holding that genuine knowledge comes from observation and measurement, that the social world can be studied with the methods of the natural sciences, and that research should seek general, testable laws. Post-positivism keeps the commitment to empirical testing while accepting that all observation is fallible and theory-laden. (Popper, 1959; Kuhn, 1962)

Research · Philosophy and paradigms
See also: Interpretivism, Critical realism, Paradigm
Read more: Positivism, Interpretivism and Pragmatism in Research Design · Ontology and Epistemology for Applied Researchers

Post-truth

Describing circumstances in which objective facts are less influential in shaping public opinion than appeals to emotion and personal belief. It was Oxford Dictionaries' word of the year for 2016. Analyses link it to declining trust in institutions, partisan media and the speed of online information. (McIntyre, 2018)

The Generation of Knowledge · Trust, quality and the future
See also: Disinformation
Read more: Post-Truth and the Crisis of Shared Knowledge

Potential outcomes also: Rubin causal model

A framework for causal inference, associated with Donald Rubin, in which each unit has a potential outcome under each possible treatment. The causal effect is the difference between them, and since only one is observed, the "fundamental problem of causal inference", effects are estimated by comparing groups under assumptions such as random assignment. (Rubin, 1974; Holland, 1986)

Statistics · Causal inference
See also: Counterfactual, Randomised controlled trial
Read more: Causal Inference: From Association to Cause

Prebunking also: inoculation

Exposing people in advance to weakened examples of misinformation techniques, with explanations, so that they are better able to recognise and resist them later, on the model of a vaccine. Online games built on this idea have reduced the perceived reliability of fake news in experiments. (Roozenbeek & van der Linden, 2019; Lewandowsky et al., 2012)

Inquiry · Ethics and limits
See also: Disinformation, Continued influence effect
Read more: Inquiry and Misinformation: Building Resistance Through Questions · Epistemic Responsibility in an Age of Misinformation

Prediction market

A market in which participants trade contracts whose payoff depends on a future event, so that prices can be read as aggregate probability estimates. Prediction markets have forecast elections and project outcomes well, though in forecasting tournaments well-aggregated team forecasts have matched or beaten them. (Wolfers & Zitzewitz, 2004; Atanasov et al., 2017)

Intelligence · Artificial and collective intelligence
See also: Wisdom of crowds, Superforecasting
Read more: Prediction Markets as Intelligence Tools

Pre-registration also: preregistration

Recording a study's hypotheses, design and analysis plan in a time-stamped public registry before data are collected or analysed. It separates confirmatory from exploratory analysis and makes p-hacking and HARKing visible. Clinical trial registration has been required by major medical journals since 2005. (Nosek et al., 2018; De Angelis et al., 2004)

Research · Research integrity
See also: Registered report, P-hacking, HARKing
Read more: Pre-Registration and Registered Reports

Principal component analysis also: PCA

A dimension-reduction method, first described by Karl Pearson in 1901, that transforms correlated variables into a smaller set of uncorrelated components that capture as much of the total variance as possible. Unlike factor analysis it does not model a latent cause of the variables. (Pearson, 1901)

Statistics · Tests and models
See also: Factor analysis
Read more: Principal Component Analysis and Dimension Reduction

Printing revolution

The transformation in the production and spread of knowledge that followed Gutenberg's movable-type press in the mid-fifteenth century. Elizabeth Eisenstein argued that standardised, multiplied and preserved texts made cumulative scholarship, the Reformation and the Scientific Revolution possible. (Eisenstein, 1979; Burke, 2000)

The Generation of Knowledge · History of knowledge
See also: Cumulative knowledge
Read more: The Printing Press and the Knowledge Revolution

Prior probability also: prior

In Bayesian reasoning, the probability assigned to a hypothesis before the current evidence is considered, often based on base rates or earlier studies. Combined with the likelihood of the evidence, it gives the posterior probability. Neglecting the prior is the source of base rate neglect. (Bayes, 1763; Jaynes, 2003)

Statistics · Bayesian and computational
See also: Bayes' theorem, Base rate neglect, Bayesian inference
Read more: Bayes' Theorem and Updating Beliefs With Evidence · Bayesian Reasoning in Everyday Inquiry

Probability sample

A sample drawn so that every unit in the population has a known, non-zero probability of selection. These probabilities allow design-based inference: unbiased estimation with weights equal to the inverse of the selection probabilities, and measurable sampling error. Simple random, systematic, stratified and cluster sampling are all forms of probability sampling. (Neyman, 1934; Kish, 1965; Lohr, 2021)

Surveys · Sampling
See also: Nonprobability sample, Stratified sampling, Cluster sampling, Sampling frame
Read more: Probability Sampling Explained: Simple Random, Systematic, Stratified and Cluster · How the 1936 Literary Digest Poll Changed Survey Sampling Forever

Problem-based learning also: PBL

An approach in which students learn by working in groups on complex, realistic problems, identifying what they need to know and researching it, with a tutor who facilitates rather than lectures. It originated in medical education at McMaster University in the late 1960s. (Hmelo-Silver, 2004)

Inquiry · Inquiry in education and learning
See also: Inquiry-based learning
Read more: Problem-Based Learning and Its Outcomes

Productive failure

A learning design in which students first attempt problems beyond their current knowledge, usually failing to solve them, before receiving instruction. Manu Kapur's studies found that this sequence can produce deeper conceptual understanding than instruction first. (Kapur, 2008)

Inquiry · Inquiry in education and learning
See also: Inquiry-based learning
Read more: The Role of Productive Failure and Iteration in Inquiry-Driven Knowledge

Propensity score

The probability that a unit receives a treatment given its observed characteristics. Rosenbaum and Rubin showed that matching, weighting or stratifying on this single score can balance the observed covariates between treated and untreated groups. It does not correct for unmeasured confounders. (Rosenbaum & Rubin, 1983)

Statistics · Causal inference
See also: Confounding, Weighting
Read more: Propensity Score Matching · Propensity Score Adjustment for Online Survey Samples

Provenance

A record of the origin and history of a piece of data or knowledge: who created it, from what sources, by what processes and with what changes. Provenance lets users judge reliability and reproduce results. The W3C PROV standard models it in terms of entities, activities and agents. (Moreau & Missier, 2013; Wilkinson et al., 2016)

The Generation of Knowledge · Trust, quality and the future
See also: Chain of custody, FAIR data
Read more: Transparency, Provenance, and the Traceability of Knowledge · Documentation as Infrastructure for Trustworthy Knowledge

Psychological safety

A shared belief within a team that it is safe to take interpersonal risks, such as asking questions, admitting mistakes or raising concerns. Amy Edmondson found that psychological safety supports team learning behaviour, which in turn improves performance. (Edmondson, 1999)

Inquiry · Inquiry in organisations
See also: Double-loop learning
Read more: Psychological Safety and the Freedom to Ask Questions

Publication bias

Systematic distortion of the published record that occurs when the chance of a study being published depends on the direction or significance of its results. It inflates average effect sizes in literature reviews and meta-analyses. Funnel plots and comparisons with registered studies are used to detect it. (Franco et al., 2014; Rosenthal, 1979)

Research · Research integrity
See also: File-drawer problem, Meta-analysis, Registered report
Read more: Publication Bias and the File-Drawer Problem

P-value

The probability, computed assuming the null hypothesis and all model assumptions are true, of obtaining a result at least as extreme as the one observed. It is not the probability that the null hypothesis is true, nor a measure of effect size. The American Statistical Association's 2016 statement warns against basing conclusions on whether p crosses 0.05. (Wasserstein & Lazar, 2016; Greenland et al., 2016)

Statistics · Inference
See also: Statistical significance, Confidence interval, Effect size, P-hacking
Read more: P-Values: What They Mean and What They Do Not · The ASA Statement on P-Values and the Move Beyond Significance

Q

Qualitative comparative analysis also: QCA

A case-oriented method developed by Charles Ragin that uses Boolean algebra and set theory to identify which combinations of conditions are sufficient or necessary for an outcome across a small to medium number of cases. It is suited to causal complexity, where different combinations of conditions can lead to the same outcome. (Ragin, 1987)

Research · Quantitative and mixed methods
See also: Case study
Read more: Configurational Comparative Methods (QCA) for Complex Causality Knowledge

Quasi-experiment

A study that tests a causal hypothesis by comparing outcomes under different conditions but without random assignment to those conditions. Designs such as non-equivalent control groups, interrupted time series and regression discontinuity use comparison groups, repeated measures and design features to rule out alternative explanations. (Shadish et al., 2002)

Research · Research design
See also: Internal validity, Natural experiment, Regression discontinuity design, Difference-in-differences
Read more: Quasi-Experimental Designs When Randomisation Is Not Possible

Question Formulation Technique also: QFT

A structured process, developed by the Right Question Institute, in which learners produce as many questions as they can about a prompt without stopping to discuss them, convert questions between open and closed forms, prioritise them and plan how to use them. (Rothstein & Santana, 2011)

Inquiry · The art of the question
Read more: Question Formulation Technique

Questionable research practices also: QRPs

Practices that fall short of fabrication or falsification but inflate false-positive rates, such as selectively reporting outcomes, deciding to collect more data after looking at results, excluding data after seeing their effect, and presenting unexpected findings as predicted. A large survey of psychologists found such practices to be common. (John et al., 2012; Simmons et al., 2011)

Research · Research integrity
See also: P-hacking, HARKing, Research misconduct
Read more: P-Hacking, HARKing and Questionable Research Practices

Quota sampling

A nonprobability method in which interviewers or online systems fill target numbers (quotas) of respondents in categories such as age, sex and region so that the sample matches the population on those variables. Selection within each quota cell is not random, so the sample can still differ from the population on unmeasured characteristics. Quota polls famously failed in the 1948 US presidential election. (Baker et al., 2013; Converse, 1987)

Surveys · Sampling
See also: Nonprobability sample, Weighting
Read more: Quota Sampling in Market Research: Practice, Rationale and Pitfalls

R

Randomised controlled trial also: RCT

An experiment in which participants are randomly allocated to an intervention or a control condition and outcomes are compared. Randomisation balances known and unknown confounders in expectation, which is why RCTs are regarded as the strongest design for estimating causal effects when they are feasible and ethical. (Shadish et al., 2002; Rubin, 1974)

Research · Research design
See also: A/B test, Confounding, Internal validity
Read more: Experimental Design: Randomisation, Control and Manipulation · Randomised Controlled Trials as the Benchmark for Causality

Randomized response technique

A method for asking sensitive questions in which a random device, unseen by the interviewer, decides whether the respondent answers the sensitive question or an innocuous one. Because no single answer reveals the respondent's status, people may answer more honestly, while the known probabilities still allow the prevalence of the sensitive trait to be estimated. (Warner, 1965; Tourangeau & Yan, 2007)

Surveys · Questionnaire design
See also: Social desirability bias
Read more: The Use of Randomized Response Techniques for Sensitive Knowledge Elicitation · Asking Sensitive Questions with Care and Precision

Realist evaluation

An approach to evaluation that asks what works, for whom, in what circumstances and why. It explains outcomes through context–mechanism–outcome configurations: programmes work by triggering mechanisms in people, and whether they do depends on context. (Pawson & Tilley, 1997)

Research · Market and business research
See also: Critical realism
Read more: Realist Evaluation Approaches for Context-Mechanism-Outcome Knowledge · Program Evaluation Research as a Distinct Mode of Applied Knowledge Generation

Red teaming also: devil's advocacy

Assigning a group to take the perspective of an adversary or to challenge a plan, so that weaknesses and alternative outcomes are exposed before they are exploited. Devil's advocacy, in which one person argues against the prevailing view, is a lighter form of the same idea. (Zenko, 2015; Heuer, 1999)

Intelligence · Analysis and tradecraft
See also: Groupthink, Structured analytic techniques
Read more: Red Teaming and Devil's Advocacy

Reflexivity

The researcher's ongoing, explicit examination of how their own background, assumptions, position and relationships shape the questions asked, the data gathered and the interpretations made. It is a standard of quality in qualitative and feminist inquiry. (Harding, 1986; Lincoln & Guba, 1985)

Inquiry · Inquiry in practice
See also: Feminist inquiry, Trustworthiness, Heuristic inquiry
Read more: Researcher Positionality and Reflexivity in Knowledge Production

Registered report

A journal article format in which the introduction, methods and analysis plan are peer reviewed before data are collected. If the protocol is accepted in principle, the journal commits to publishing the results whatever they show, which removes the incentive to produce positive findings. Registered reports report far fewer positive results than standard articles. (Chambers, 2013; Scheel et al., 2021)

Research · Research integrity
See also: Pre-registration, Publication bias
Read more: Pre-Registration and Registered Reports

Regression

A family of methods that model how the expected value of an outcome changes with one or more predictors. In linear regression, coefficients estimate the change in the outcome per unit change in a predictor, holding the other predictors constant. Galton coined the term from his observation of regression to the mean in heredity. (Galton, 1886; Gelman & Hill, 2007)

Statistics · Tests and models
See also: Logistic regression, Generalised linear model, Confounding
Read more: Simple and Multiple Linear Regression

Regression discontinuity design also: RDD

A quasi-experimental design used when treatment is assigned by whether a score crosses a cut-off, such as a test mark for a scholarship. Comparing units just above and just below the threshold estimates the causal effect at the cut-off, because those units are otherwise similar. (Thistlethwaite & Campbell, 1960; Angrist & Pischke, 2009)

Statistics · Causal inference
See also: Quasi-experiment, Instrumental variable
Read more: Regression Discontinuity Designs

Regression to the mean

The tendency for unusually high or low measurements to be followed by values closer to the average, because extreme values partly reflect chance. Francis Galton described it in heredity. Ignoring it leads people to credit interventions, or blame them, for changes that would have happened anyway. (Galton, 1886; Kahneman & Tversky, 1973)

Statistics · Communication and misuse
See also: Selection bias
Read more: Regression to the Mean in Business Performance

Reliability

The consistency of a measurement: the degree to which it gives the same result under the same conditions. Forms include test–retest reliability (stability over time), internal consistency (agreement among items, often measured by Cronbach's alpha) and inter-rater reliability (agreement among coders). A measure can be reliable without being valid. (Cronbach, 1951)

Research · Quantitative and mixed methods
See also: Construct validity, Cronbach's alpha, Inter-rater reliability
Read more: Reliability and Validity: The Twin Pillars of Measurement · Reliability Statistics: Cronbach's Alpha and Alternatives

Replication crisis

The finding, from large-scale replication projects in psychology, economics, medicine and other fields since about 2011, that many published results cannot be reproduced. In one project only about 36% of 100 psychology studies produced a significant result again. Causes include small samples, publication bias and flexible analysis; reforms include pre-registration and open data. (Open Science Collaboration, 2015; Ioannidis, 2005; Munafò et al., 2017)

Research · Research integrity
See also: P-hacking, Publication bias, Pre-registration, Open science
Read more: The Replication Crisis: Causes, Evidence and Reforms · Research on Research (Meta-Research) for Improving Knowledge Generation Practices

Research misconduct also: FFP

Fabrication (making up data), falsification (manipulating data, materials or processes) or plagiarism (using others' work without credit) in proposing, performing, reviewing or reporting research. A meta-analysis of surveys found that about 2% of scientists admitted to having fabricated or falsified data at least once. (Fanelli, 2009)

Research · Research integrity
See also: Questionable research practices, Peer review
Read more: Research Misconduct: Fabrication, Falsification and Plagiarism · Scientific Misconduct Investigations

Respondent-driven sampling also: RDS

A network-based method for sampling hidden or hard-to-reach populations, in which initial participants (seeds) recruit peers, who recruit further peers in successive waves. Recruitment is tracked and weighted by participants' network sizes to reduce the bias of chain referral, under assumptions that are often hard to verify in practice. (Heckathorn, 1997)

Surveys · Sampling
See also: Nonprobability sample, Snowball sampling
Read more: Respondent-Driven Sampling and Network-Based Recruitment · Sampling Rare and Hard-to-Reach Populations

Response rate

The proportion of eligible sample members who complete a survey. AAPOR's Standard Definitions set out several formulas that differ in how partial interviews and cases of unknown eligibility are treated. A low response rate raises the risk of nonresponse bias but does not by itself show that bias is present. (American Association for Public Opinion Research, 2023; Groves & Peytcheva, 2008)

Surveys · Nonresponse, weighting and data quality
See also: Nonresponse bias, Unit nonresponse
Read more: The Decline in Survey Response Rates and What It Means for Accuracy · Establishment of Survey Quality Indicators Beyond Traditional Response Rates

Responsive design

A survey design that sets out in advance how fieldwork will change in phases, based on paradata and early results, to control costs and errors; for example switching mode or raising incentives for groups with low response. Adaptive design is a related approach that tailors protocols to sample members from the start. (Groves & Heeringa, 2006)

Surveys · Nonresponse, weighting and data quality
See also: Paradata, Nonresponse bias
Read more: Adaptive and Responsive Survey Designs for Efficient Knowledge Acquisition

Restructuring

The Gestalt psychologists' term for the change in how a problem is represented that produces insight, for example seeing an object's unusual use. Karl Duncker's studies of functional fixedness, such as the candle problem, showed how prior use of an object blocks restructuring. (Duncker, 1945; Kounios & Beeman, 2014)

Insights · What an insight is
See also: Insight, Aha moment
Read more: Gestalt Psychology and Restructuring in Problem-Solving

Root cause analysis also: RCA

A structured investigation of an incident or problem that seeks the underlying causes, in processes, systems and organisation, rather than stopping at the immediate error. Methods include the five whys, fishbone diagrams, fault tree analysis and timeline reconstruction. (Reason, 2000; Ohno, 1988)

Investigation · Scientific and technical investigation
See also: Five whys, Fishbone diagram, Swiss cheese model
Read more: Root Cause Analysis Methods: Fishbone, Fault Tree and Five Whys · Root Cause Investigation of Customer Complaints

S

Sampling distribution

The distribution of a statistic, such as a sample mean, across all possible samples of the same size from a population. Its standard deviation is the standard error. It is the basis of confidence intervals and hypothesis tests. (Lohr, 2021)

Statistics · Distributions
See also: Standard error, Central limit theorem, Bootstrap
Read more: Sampling Distributions and Standard Errors

Sampling frame

The list or procedure used to identify and select members of the target population, such as an address file, a telephone number range or a customer database. The quality of a sample can be no better than its frame: units missing from the frame cannot be selected, which leads to coverage error. (Groves et al., 2009; Lohr, 2021)

Surveys · Sampling
See also: Coverage error, Probability sample
Read more: Sampling Frames and Coverage Error: Who Is Missing From Your List? · Multi-Frame Sampling Designs for Improved Coverage of Target Populations

Satisficing

Giving answers that are merely acceptable rather than optimal, because carefully completing each step of the response process is demanding. Weak satisficing includes choosing the first reasonable option or agreeing with statements; strong satisficing includes straightlining, choosing "don't know" or random answering. It is more likely when the task is difficult, motivation is low and ability is limited. (Krosnick, 1991)

Surveys · Nonresponse, weighting and data quality
See also: Straightlining, Acquiescence bias, Survey response process
Read more: Satisficing in Surveys: Why Respondents Take Mental Shortcuts

Saturation

The point in qualitative data collection at which new interviews or observations no longer produce new themes or insights relevant to the research question. In one widely cited study, most themes appeared within the first twelve interviews in a fairly homogeneous group, though the number needed depends on the aims, population and analysis. (Guest et al., 2006; Glaser & Strauss, 1967)

Research · Qualitative research
See also: Grounded theory, Thematic analysis
Read more: Saturation in Qualitative Research: How Many Interviews Are Enough?

Say–do gap also: attitude–behaviour gap

The difference between what people say they think, intend or will do and what they actually do. Meta-analyses show that intentions explain only part of the variation in behaviour, and LaPiere's 1934 study found that stated refusals to serve Chinese guests contrasted sharply with actual service. Insight work therefore combines stated and observed data. (LaPiere, 1934; Sheeran, 2002)

Insights · Generating insight
See also: Social desirability bias, Implicit Association Test
Read more: The Say-Do Gap: When Stated Attitudes Diverge From Behaviour

Scenario planning

A method for exploring several plausible futures, built from the key uncertainties and driving forces affecting an organisation, so that strategies can be tested against each. Its aim is to widen thinking and prepare for change rather than to predict a single outcome. (Schoemaker, 1995)

Intelligence · Analysis and tradecraft
See also: Weak signal, Indicators and warnings
Read more: Scenario Planning: Preparing for Several Futures

Scientific Revolution

The transformation of natural knowledge in Europe between roughly the mid-sixteenth and late seventeenth centuries, associated with Copernicus, Galileo, Kepler, Bacon, Descartes, Boyle and Newton, and with new institutions such as the Royal Society. Historians now stress its gradual, contested and social character. (Shapin, 1996)

The Generation of Knowledge · History of knowledge
See also: Paradigm, Norms of science
Read more: The Scientific Revolution and the Birth of Modern Knowledge · The Royal Society and the Institutionalisation of Science

Scoping review

A review that maps the extent, range and nature of research on a broad topic, identifies key concepts and gaps, and often precedes a systematic review. It follows a transparent search process but does not usually appraise study quality or pool results. (Arksey & O'Malley, 2005)

Research · Literature and evidence
See also: Systematic review
Read more: Scoping Reviews and Rapid Evidence Assessments

SECI model

Ikujiro Nonaka's model of organisational knowledge creation as a spiral of four conversions between tacit and explicit knowledge: socialisation (tacit to tacit), externalisation (tacit to explicit), combination (explicit to explicit) and internalisation (explicit to tacit). (Nonaka, 1994; Nonaka & Takeuchi, 1995)

The Generation of Knowledge · Knowledge management
See also: Tacit knowledge, Explicit knowledge, Knowledge management
Read more: Nonaka's SECI Model of Organisational Knowledge Creation · The Knowledge-Creating Company

Segmentation also: market segmentation

Dividing a heterogeneous market into smaller groups of customers with similar needs, characteristics or behaviour, so that products and marketing can be targeted to each. Wendell Smith introduced the concept in 1956. Segments are commonly derived from survey data with cluster or latent class analysis. (Smith, 1956)

Research · Market and business research
See also: Cluster analysis, Latent class analysis, Persona
Read more: Segmentation Research: Dividing Markets Into Meaningful Groups · Segmentation as a Source of Insight · Cluster Analysis for Market Segmentation

Selection bias

Bias that arises when the units included in an analysis differ systematically from the population of interest because of how they were selected, or because selection depends on both exposure and outcome. Coverage error, nonresponse bias, survivorship bias and collider bias are forms of it. (Hernán et al., 2004)

Statistics · Causal inference
See also: Collider bias, Survivorship bias, Nonresponse bias
Read more: Survivorship Bias in Data and Decisions · Correlation Is Not Causation: Spurious Relationships in Data

Semiotics

The study of signs and how they create meaning. In consumer research it analyses how products, packaging, advertising and brands communicate through codes, symbols and cultural conventions, and how consumers read them. (Mick, 1986)

Insights · Generating insight
Read more: Cultural Insight and Semiotics

Sensemaking

Karl Weick's term for the ongoing process by which people and organisations give meaning to ambiguous or surprising events, by noticing cues, placing them in frameworks and acting on plausible interpretations. It emphasises that understanding is constructed in action and in conversation, not simply discovered in data. (Weick, 1995)

Insights · What an insight is
See also: Insight
Read more: Sensemaking Processes and the Emergence of Interpretive Knowledge

Serendipity

Discovery by a combination of accident and sagacity: finding something valuable that was not sought, and recognising its significance. Horace Walpole coined the word in 1754; Robert Merton studied its role in science, emphasising the prepared mind needed to notice an anomaly. (Merton & Barber, 2004)

The Generation of Knowledge · Knowledge production
See also: Apophenia, Aha moment
Read more: Serendipity in the Generation of Knowledge

Simpson's paradox

A reversal in which an association between two variables in every subgroup disappears or reverses when the subgroups are combined. The Berkeley graduate admissions data are a classic example. Which comparison is correct depends on the causal structure, not on the arithmetic. (Simpson, 1951; Bickel et al., 1975; Pearl, 2009)

Statistics · Communication and misuse
See also: Confounding, Ecological fallacy
Read more: Simpson's Paradox: When Aggregated Data Mislead · Correlation Is Not Causation: Spurious Relationships in Data

Small area estimation

Methods for producing reliable estimates for geographic areas or groups where the survey sample is too small for direct estimates. Models borrow strength from other areas and from auxiliary data such as census or administrative records. (Rao & Molina, 2015)

Statistics · Survey statistics
See also: MRP, Multilevel model
Read more: Small Area Estimation

Snowball sampling also: chain-referral sampling

A nonprobability method in which initial participants refer others they know who meet the study criteria, who in turn refer more. It is useful for reaching hidden or rare populations, but the sample reflects the social networks of the starting points. Respondent-driven sampling adds tracking and weighting to reduce this bias. (Heckathorn, 1997)

Surveys · Sampling
See also: Respondent-driven sampling, Nonprobability sample
Read more: Non-Probability Sampling: Convenience, Quota, Snowball and Purposive Methods · Sampling Rare and Hard-to-Reach Populations

Social desirability bias

The tendency to answer in ways that present oneself favourably, over-reporting approved behaviours such as voting and under-reporting disapproved ones such as drug use. It is strongest with interviewers present and on sensitive topics. Self-administration, assurances of confidentiality and indirect techniques such as randomized response reduce it. (Tourangeau & Yan, 2007)

Surveys · Nonresponse, weighting and data quality
See also: Randomized response technique, Mode effect, Say–do gap
Read more: Social Desirability Bias and Techniques for Reducing It · Asking Sensitive Questions with Care and Precision

Social epistemology

The study of the social dimensions of knowledge: how testimony, institutions, expertise, communication systems and collective deliberation help or hinder groups and individuals in forming true beliefs. Alvin Goldman framed it as evaluating social practices by their tendency to produce truth. (Goldman, 1999)

The Generation of Knowledge · Epistemology
See also: Testimony, Epistemic injustice, Wisdom of crowds
Read more: Social Epistemology: Knowledge as a Collective Achievement · Knowledge From Many Voices: The Epistemology of Aggregated Opinion

Socratic questioning

A disciplined method of asking questions to examine ideas, uncover assumptions, test the logic and evidence behind beliefs and follow their implications. Derived from the dialogues of Plato's Socrates, it is used in teaching, coaching, therapy and analysis. (Paul & Elder, 2006)

Inquiry · Philosophy of inquiry
See also: Critical thinking, Five whys
Read more: Socratic Questioning and the Examined Life

Source evaluation

Assessing information by the reliability of its source and the credibility of its content, often with a two-part grading such as the NATO Admiralty code (A–F for source reliability, 1–6 for information credibility). Research shows such ratings are often inconsistent and that analysts should consider a source's access and corroboration explicitly. (Irwin & Mandel, 2019)

Intelligence · Analysis and tradecraft
See also: Corroboration, Open-source intelligence, Deception
Read more: Source Evaluation: Reliability, Access, and Corroboration

Spurious correlation

An association between two variables that does not reflect a causal link between them, because both are driven by a third factor, because of chance, because of shared trends over time, or because of how the variables were constructed. Karl Pearson used the term in 1897 for correlations between ratios with a common denominator. (Pearson, 1897; Yule, 1926; Messerli, 2012)

Statistics · Communication and misuse
See also: Confounding, Correlation coefficient
Read more: Correlation Is Not Causation: Spurious Relationships in Data

Standard deviation

A measure of spread equal to the square root of the variance, the average squared distance of values from their mean. It is expressed in the units of the data. For roughly normal data, about two-thirds of values lie within one standard deviation of the mean. (Tukey, 1977)

Statistics · Descriptive statistics
See also: Normal distribution, Standard error
Read more: Measures of Dispersion: Range, Variance and Standard Deviation

Standard error

The standard deviation of the sampling distribution of a statistic, measuring how much the statistic would vary from sample to sample. For a mean from a simple random sample it equals the standard deviation divided by the square root of the sample size. Complex designs require design-based variance estimation. (Lohr, 2021; Kish, 1965)

Statistics · Distributions
See also: Sampling distribution, Confidence interval, Design effect
Read more: Sampling Distributions and Standard Errors · Variance Estimation in Complex Survey Samples

Statistical power

The probability that a test will detect an effect of a given size if it exists, that is, one minus the probability of a Type II error. Power depends on the effect size, the sample size, the significance level and the variability of the data. Underpowered studies miss real effects and exaggerate the effects they do find. (Cohen, 1988)

Statistics · Inference
See also: Type I error, Effect size, Type I error
Read more: Type I and Type II Errors and Statistical Power · Power Analysis for Planning Studies

Statistical process control also: SPC

Monitoring a process over time with control charts that separate common-cause variation, inherent in a stable process, from special-cause variation that signals a change worth investigating. Walter Shewhart developed the method at Bell Laboratories in the 1920s. (Shewhart, 1931)

Statistics · Communication and misuse
Read more: Statistical Process Control and Monitoring for Ongoing Knowledge Quality

Statistical significance

The label applied when a p-value falls below a chosen threshold, conventionally 0.05. A significant result is not necessarily important, and a non-significant result is not evidence of no effect. Many statisticians recommend reporting effect sizes and confidence intervals instead of dichotomising results. (Wasserstein & Lazar, 2016; Greenland et al., 2016)

Statistics · Inference
See also: P-value, Effect size, Type I error
Read more: Statistical Significance Versus Practical Significance · The ASA Statement on P-Values and the Move Beyond Significance

Storytelling with data

Presenting analytical results as a narrative with a clear audience, message and structure, supported by visuals designed to make the key point obvious. The aim is to help decision-makers understand and remember what the data mean, without distorting the evidence. (Knaflic, 2015; Tufte, 1983)

Insights · Communicating insight
Read more: Storytelling With Data: Narrative Structures for Insight · Visualising Insights for Different Audiences

Straightlining also: nondifferentiation

Giving the same answer to every item in a grid or battery of questions regardless of content. It is a common indicator of satisficing or careless responding, although identical answers can sometimes be genuine. Shorter grids, item-by-item presentation and varied item wording reduce it. (Krosnick, 1991; Meade & Craig, 2012)

Surveys · Nonresponse, weighting and data quality
See also: Satisficing, Careless responding
Read more: Matrix Questions and Straightlining: Efficiency Versus Data Quality · Speeders, Straightliners and Bots: Cleaning Online Survey Data

Stratified sampling

A probability design in which the population is divided into non-overlapping groups (strata) and a separate sample is drawn from each. Stratification guarantees representation of each stratum and, when strata differ on the survey variable, increases precision. Neyman (1934) showed how to allocate the sample across strata to minimise variance. (Neyman, 1934; Lohr, 2021)

Surveys · Sampling
See also: Probability sample, Design effect
Read more: Stratified Sampling for Business Research: Gains in Precision and Efficiency · Probability Sampling Explained: Simple Random, Systematic, Stratified and Cluster

Structured analytic techniques also: SATs

Methods that make an analyst's reasoning explicit and open to review, such as analysis of competing hypotheses, key assumptions checks, red teaming, indicators and scenarios. They were promoted to counter the cognitive biases Heuer described, although evidence on how much they improve accuracy is mixed. (Heuer, 1999)

Intelligence · Analysis and tradecraft
See also: Analysis of competing hypotheses, Key assumptions check, Heuristics and biases
Read more: Structured Analytic Techniques in Intelligence Analysis · Cognitive Bias Mitigation Training and Its Measurable Effects on Analytic Knowledge

Superforecasting

The practice of the top performers in the Good Judgment Project, a forecasting tournament run from 2011 to 2015, whose probability forecasts on geopolitical questions were markedly more accurate than those of other participants. Accuracy was linked to cognitive style, training in probabilistic reasoning, teamwork and frequent small updates. (Tetlock & Gardner, 2015; Mellers et al., 2015)

Intelligence · Analysis and tradecraft
See also: Brier score, Calibration, Wisdom of crowds
Read more: Superforecasting: What Research Says About Accurate Prediction

Survey response process

The model of answering a survey question as four components: comprehension of the question, retrieval of relevant information from memory, judgement to combine that information into an answer, and response, mapping the answer onto the categories offered. Errors can enter at each stage; the model underpins cognitive interviewing and much questionnaire design advice. (Tourangeau et al., 2000)

Surveys · Questionnaire design
See also: Cognitive interviewing, Satisficing, Telescoping
Read more: The Psychology of Answering a Survey Question: Comprehension, Retrieval, Judgement and Response

Survivorship bias

Drawing conclusions only from cases that passed a selection process, such as successful firms or returning aircraft, while overlooking those that did not. Abraham Wald's Second World War analysis of where to armour bombers, based on where surviving planes were not hit, is the classic example. (Mangel & Samaniego, 1984)

Statistics · Communication and misuse
See also: Selection bias, Collider bias
Read more: Survivorship Bias in Data and Decisions

Swiss cheese model

James Reason's model of accident causation in which an organisation's defences are slices of cheese with holes (active failures and latent conditions). An accident happens when holes in successive layers line up. It directs investigators towards system weaknesses rather than individual blame. (Reason, 2000; Reason, 1990)

Investigation · Scientific and technical investigation
See also: Root cause analysis
Read more: Accident Investigation: Lessons From Aviation Safety

SWOT analysis

A planning framework that lists an organisation's internal strengths and weaknesses and its external opportunities and threats. It is easy to use and communicate but is often criticised for producing unprioritised lists; it is most useful when each item is backed by evidence and linked to decisions. (Porter, 1979)

Intelligence · Market and competitive intelligence
See also: Five forces, Competitive intelligence
Read more: SWOT and PESTLE Analysis: Uses and Limitations

Synthetic control method

A method for estimating the effect of an intervention on a single unit, such as a region, by constructing a weighted combination of untreated units that closely matches the treated unit before the intervention. The divergence between the treated unit and its synthetic control afterwards estimates the effect. (Abadie et al., 2010)

Statistics · Causal inference
See also: Difference-in-differences, Counterfactual
Read more: Synthetic Control Methods for Comparative Case Knowledge

Systematic review

A review that answers a defined question by identifying, appraising and synthesising all relevant studies using explicit, reproducible methods: a protocol, comprehensive searches, predefined inclusion criteria and risk-of-bias assessment. The PRISMA 2020 statement sets out how such reviews should be reported. (Page et al., 2021)

Research · Literature and evidence
See also: Meta-analysis, Scoping review
Read more: Systematic Reviews: Protocols, Searches and Synthesis

Systems thinking

An approach to inquiry that looks at how the parts of a system interact through stocks, flows, feedback loops and delays to produce behaviour over time, rather than examining parts in isolation. It helps explain why interventions in complex systems often have unintended effects. (Meadows, 2008)

Inquiry · Logic and reasoning
See also: Wicked problem
Read more: Systems Thinking Tools for Complex Inquiry Problems · Insights in Complex Systems: Emergence, Feedback, and Knowledge Dynamics

T

Tacit knowledge

Knowledge that people have but cannot fully put into words, such as how to ride a bicycle or recognise a face. Michael Polanyi summarised it as "we can know more than we can tell". Tacit knowledge is transferred mainly by shared practice and is hard to capture in documents. (Polanyi, 1966; Nonaka, 1994)

The Generation of Knowledge · Epistemology
See also: Explicit knowledge, SECI model, Know-how
Read more: Tacit and Explicit Knowledge: Polanyi's Distinction

Telescoping

A recall error in which respondents report events as having happened more recently (forward telescoping) or longer ago (backward telescoping) than they did, moving them into or out of the reference period of a question. Bounded interviewing, where a previous interview marks the start of the period, was developed to reduce it. (Neter & Waksberg, 1964; Tourangeau et al., 2000)

Surveys · Questionnaire design
See also: Survey response process
Read more: Recall Bias and Telescoping in Retrospective Survey Questions

Testimony

Knowledge gained from what others tell us, through conversation, books, experts, surveys and media. Most of what any person knows depends on it. Philosophers debate whether testimony is a basic source of justification or must be backed by evidence of the speaker's reliability. (Coady, 1992; Goldman, 1999)

The Generation of Knowledge · Epistemology
See also: Social epistemology
Read more: Testimony as a Source of Knowledge

Thematic analysis

A method for identifying, analysing and reporting patterns of meaning (themes) in qualitative data. Braun and Clarke's widely used version has six phases: familiarisation, generating codes, constructing themes, reviewing themes, defining and naming themes, and writing up. (Braun & Clarke, 2006)

Research · Qualitative research
See also: Grounded theory, Coding, Content analysis
Read more: Thematic Analysis: A Step-by-Step Method · Coding Qualitative Data: Manual and Software-Assisted Approaches

Thought experiment

An imagined scenario used to test a theory, clarify a concept or reveal a hidden assumption by reasoning about what would happen, as in Galileo's falling bodies or Einstein's train. Thought experiments are a form of inquiry that relies on reasoning rather than new observation. (Sorensen, 1992)

Inquiry · Logic and reasoning
See also: Counterfactual
Read more: Thought Experiments as Tools of Inquiry

Total survey error also: TSE

A framework that treats survey quality as the sum of all errors affecting an estimate: coverage, sampling, nonresponse and adjustment errors on the representation side, and validity, measurement and processing errors on the measurement side. It encourages designers to balance errors against costs rather than focusing on sampling error alone. (Groves & Lyberg, 2010; Groves et al., 2009)

Surveys · Foundations and history
See also: Coverage error, Nonresponse bias, Margin of error, Measurement error
Read more: The Total Survey Error Framework: A Unified Model of Survey Quality

Triangulation

Using more than one method, data source, investigator or theory to study the same phenomenon, so that conclusions do not depend on the weaknesses of a single approach. Norman Denzin distinguished data, investigator, theory and methodological triangulation. (Denzin, 1978)

Research · Quantitative and mixed methods
See also: Mixed methods research, Corroboration
Read more: Triangulation: Strengthening Conclusions With Multiple Methods

Triple helix

A model of innovation in which universities, industry and government interact, each taking on some roles of the others, for example universities founding companies. Etzkowitz and Leydesdorff proposed it as an alternative to models centred on firms or on national systems. (Etzkowitz & Leydesdorff, 2000)

The Generation of Knowledge · Knowledge production
See also: Mode 2 knowledge production, Knowledge economy
Read more: The Triple Helix: Universities, Industry and Government

Trustworthiness

Lincoln and Guba's criteria for judging qualitative research, parallel to validity and reliability in quantitative work: credibility (confidence in the findings), transferability (applicability to other contexts), dependability (consistency of the process) and confirmability (grounding in the data rather than the researcher's bias). (Lincoln & Guba, 1985)

Research · Qualitative research
See also: Saturation, Member checking, Reflexivity
Read more: Trustworthiness in Qualitative Research: Credibility, Transferability, Dependability · The Use of Member Checking and Respondent Validation for Knowledge Confirmation

t-test also: Student's t-test

A test comparing a mean with a value, or two means with each other, using the t distribution, which accounts for the extra uncertainty from estimating the standard deviation in small samples. William Sealy Gosset published it in 1908 under the pseudonym "Student". (Student, 1908)

Statistics · Tests and models
See also: P-value, Effect size
Read more: The t-Test: Comparing Two Means

Tunnel vision

In criminal justice, the process by which investigators focus on a suspect early and then select and filter evidence in ways that build a case against that person while ignoring evidence pointing elsewhere. It is a leading contributor to wrongful convictions and is driven by confirmation bias, hindsight bias and institutional pressures. (Findley & Scott, 2006)

Investigation · Foundations
See also: Confirmation bias in investigation
Read more: Confirmation Bias and Tunnel Vision in Investigations

Type I error

Rejecting a null hypothesis that is true, a false positive. Its probability is set by the significance level, often 0.05. A Type II error is failing to reject a false null hypothesis, a false negative; its probability is one minus the power. (Cohen, 1988)

Statistics · Inference
See also: Statistical power, Multiple comparisons, False discovery rate
Read more: Type I and Type II Errors and Statistical Power

U

Unit nonresponse

The failure to obtain any survey data from a sampled unit, because the person or organisation could not be contacted, refused, or was unable to take part. It is distinguished from item nonresponse and is the main source of declining response rates in household surveys. (Groves & Couper, 1998; Brick, 2013)

Surveys · Nonresponse, weighting and data quality
See also: Item nonresponse, Response rate, Nonresponse bias
Read more: Unit Nonresponse Versus Item Nonresponse: Causes and Remedies

V

Value of information

In decision theory, the increase in expected value of a decision that new information would bring, compared with deciding without it. Ronald Howard formalised it in 1966. It shows that information is worth buying only if it could change the decision, and that the expected value of perfect information sets an upper limit on what to pay for research. (Howard, 1966)

The Generation of Knowledge · Knowledge and decisions
See also: Decision-making under uncertainty, Information asymmetry
Read more: The Value of Information in Decision Theory

Voice of the customer also: VoC

A structured process for capturing customers' needs and preferences in their own words, through interviews, surveys, complaints and feedback, and translating them into product and service requirements. Griffin and Hauser's 1993 study showed that a modest number of interviews uncovers most customer needs. (Griffin & Hauser, 1993)

Insights · Customer and market insight
See also: Customer experience, Net Promoter Score
Read more: Voice of the Customer Programmes

W

Wallas's four stages

Graham Wallas's 1926 model of creative thought as preparation (working on the problem), incubation (setting it aside), illumination (the solution appears) and verification (testing it). The model remains a common framework for research on insight and creativity. (Wallas, 1926)

Insights · What an insight is
See also: Incubation, Aha moment
Read more: Wallas's Four Stages of Creative Thought

Weak signal

An early, imprecise indication of a change that may become significant. Igor Ansoff argued in 1975 that firms should respond in graduated steps to weak signals of strategic surprise rather than wait for full information. Horizon scanning is the systematic search for such signals. (Ansoff, 1975)

Intelligence · Analysis and tradecraft
See also: Indicators and warnings, Scenario planning
Read more: Weak Signals and Horizon Scanning

Weighted mean

An average in which each value counts in proportion to a weight, calculated as the sum of each value times its weight divided by the sum of the weights. In surveys the weights represent how many population members each respondent stands for, so the weighted mean estimates the population mean. (Horvitz & Thompson, 1952; Kish, 1992)

Statistics · Descriptive statistics
See also: Weighting, Effective sample size
Read more: Weighted Means and Weighted Percentages · Statistical Analysis of Weighted Survey Data

Weighting also: survey weights

Assigning each respondent a weight so that weighted estimates represent the target population. Base weights correct for unequal selection probabilities; nonresponse and calibration adjustments (post-stratification, raking, generalised regression) make weighted totals match known population figures. Weighting can reduce bias but usually increases variance. (Kalton & Flores-Cervantes, 2003; Deville & Särndal, 1992; Horvitz & Thompson, 1952)

Surveys · Nonresponse, weighting and data quality
See also: Design effect, Effective sample size, Nonresponse bias, Weighted mean
Read more: Survey Weighting Methods: Post-Stratification, Raking and Calibration · Weighted Means and Weighted Percentages · Statistical Analysis of Weighted Survey Data

Whistleblowing

The disclosure by members of an organisation of illegal, immoral or illegitimate practices under the control of their employers to people or bodies that may be able to act. Internal reporting channels, protection from retaliation and proper investigation of reports determine whether whistleblowing helps an organisation correct wrongdoing. (Near & Miceli, 1985)

Investigation · Business and corporate investigation
Read more: Whistleblowing and the Investigation of Internal Reports

Wicked problem

Horst Rittel and Melvin Webber's 1973 term for social and policy problems that have no definitive formulation, no stopping rule, no true-or-false solutions and no way to test solutions without consequences, such as poverty or climate policy. Such problems call for iterative inquiry and stakeholder involvement rather than a single optimal solution. (Rittel & Webber, 1973)

Inquiry · Logic and reasoning
See also: Systems thinking
Read more: Inquiry Processes for Wicked Problems and Ill-Structured Domains

Wisdom of crowds

The finding that the aggregate of many independent judgements is often more accurate than most individual judgements, as in Francis Galton's 1907 report that the median guess of an ox's weight at a fair was within 1% of the true value. It depends on diversity and independence of opinions; when judgements are correlated or influenced by each other the advantage shrinks. (Galton, 1907; Surowiecki, 2004)

Intelligence · Artificial and collective intelligence
See also: Collective intelligence, Prediction market
Read more: Collective Intelligence and the Wisdom of Crowds · When Crowds Are Wise and When They Are Not · Knowledge From Many Voices: The Epistemology of Aggregated Opinion

Words of estimative probability also: WEP

Verbal expressions such as "likely" or "almost certain" used in intelligence estimates to convey probability. Sherman Kent showed in 1964 that readers interpret such words very differently and proposed linking each to a numerical range. Many intelligence agencies now publish standard probability yardsticks. (Kent, 1964; Mandel & Barnes, 2014)

Intelligence · Analysis and tradecraft
See also: Calibration
Read more: Expressing Probability in Intelligence Estimates: Words of Estimative Probability

References

Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic control methods for comparative case studies: Estimating the effect of California's tobacco control program. Journal of the American Statistical Association, 105(490), 493–505. https://doi.org/10.1198/jasa.2009.ap08746

Ackoff, R. L. (1989). From data to wisdom. Journal of Applied Systems Analysis, 16, 3–9.

Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/TAC.1974.1100705

Akerlof, G. A. (1970). The market for "lemons": Quality uncertainty and the market mechanism. The Quarterly Journal of Economics, 84(3), 488–500. https://doi.org/10.2307/1879431

Alavi, M., & Leidner, D. E. (2001). Review: Knowledge management and knowledge management systems: Conceptual foundations and research issues. MIS Quarterly, 25(1), 107–136. https://doi.org/10.2307/3250961

American Association for Public Opinion Research. (2023). Standard definitions: Final dispositions of case codes and outcome rates for surveys (10th ed.). https://aapor.org/wp-content/uploads/2023/05/Standards-Definitions-10th-edition.pdf

Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. Longman.

Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press. https://doi.org/10.1515/9781400829828

Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434), 444–455. https://doi.org/10.1080/01621459.1996.10476902

Anscombe, F. J. (1973). Graphs in statistical analysis. The American Statistician, 27(1), 17–21. https://doi.org/10.1080/00031305.1973.10478966

Ansoff, H. I. (1975). Managing strategic surprise by response to weak signals. California Management Review, 18(2), 21–33. https://doi.org/10.2307/41164635

Argyris, C., & Schön, D. A. (1978). Organizational learning: A theory of action perspective. Addison-Wesley.

Arksey, H., & O'Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32. https://doi.org/10.1080/1364557032000119616

Arrow, K. J. (1962). Economic welfare and the allocation of resources for invention. In National Bureau of Economic Research (Ed.), The rate and direction of inventive activity: Economic and social factors (pp. 609–626). Princeton University Press.

Association of Certified Fraud Examiners. (2024). Occupational fraud 2024: A report to the nations. https://www.acfe.com/RTTN

Association of College and Research Libraries. (2015). Framework for information literacy for higher education. American Library Association. https://www.ala.org/acrl/standards/ilframework

Atanasov, P., Rescober, P., Stone, E., Swift, S. A., Servan-Schreiber, E., Tetlock, P., Ungar, L., & Mellers, B. (2017). Distilling the wisdom of crowds: Prediction markets vs. prediction polls. Management Science, 63(3), 691–706. https://doi.org/10.1287/mnsc.2015.2374

Baker, R., Brick, J. M., Bates, N. A., Battaglia, M., Couper, M. P., Dever, J. A., Gile, K. J., & Tourangeau, R. (2013). Summary report of the AAPOR task force on non-probability sampling. Journal of Survey Statistics and Methodology, 1(2), 90–143. https://doi.org/10.1093/jssam/smt008

Bar-Hillel, M. (1980). The base-rate fallacy in probability judgments. Acta Psychologica, 44(3), 211–233. https://doi.org/10.1016/0001-6918(80)90046-3

Barney, J. (1991). Firm resources and sustained competitive advantage. Journal of Management, 17(1), 99–120. https://doi.org/10.1177/014920639101700108

Bayes, T. (1763). An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 53, 370–418. https://doi.org/10.1098/rstl.1763.0053

Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610–623). ACM. https://doi.org/10.1145/3442188.3445922

Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x

Berkson, J. (1946). Limitations of the application of fourfold table analysis to hospital data. Biometrics Bulletin, 2(3), 47–53. https://doi.org/10.2307/3002000

Betts, R. K. (1978). Analysis, war, and decision: Why intelligence failures are inevitable. World Politics, 31(1), 61–89. https://doi.org/10.2307/2009967

Bickel, P. J., Hammel, E. A., & O'Connell, J. W. (1975). Sex bias in graduate admissions: Data from Berkeley. Science, 187(4175), 398–404. https://doi.org/10.1126/science.187.4175.398

BonJour, L. (1985). The structure of empirical knowledge. Harvard University Press.

Bonney, R., Cooper, C. B., Dickinson, J., Kelling, S., Phillips, T., Rosenberg, K. V., & Shirk, J. (2009). Citizen science: A developing tool for expanding science knowledge and scientific literacy. BioScience, 59(11), 977–984. https://doi.org/10.1525/bio.2009.59.11.9

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Wiley. https://doi.org/10.1002/9780470743386

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Brick, J. M. (2013). Unit nonresponse and weighting adjustments: A critical review. Journal of Official Statistics, 29(3), 329–353. https://doi.org/10.2478/jos-2013-0026

Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3. https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2

Brown, T. (2008). Design thinking. Harvard Business Review, 86(6), 84–92.

Burke, P. (2000). A social history of knowledge: From Gutenberg to Diderot. Polity.

Card, D., & Krueger, A. B. (1994). Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania. American Economic Review, 84(4), 772–793.

Casey, E. (2011). Digital evidence and computer crime: Forensic science, computers and the internet (3rd ed.). Academic Press.

Chambers, C. D. (2013). Registered Reports: A new publishing initiative at Cortex. Cortex, 49(3), 609–610. https://doi.org/10.1016/j.cortex.2012.12.016

Chen, H., Chiang, R. H. L., & Storey, V. C. (2012). Business intelligence and analytics: From big data to big impact. MIS Quarterly, 36(4), 1165–1188. https://doi.org/10.2307/41703503

Christensen, C. M., Hall, T., Dillon, K., & Duncan, D. S. (2016). Know your customers' "jobs to be done." Harvard Business Review, 94(9), 54–62.

Coady, C. A. J. (1992). Testimony: A philosophical study. Clarendon Press.

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.

Cohen, W. M., & Levinthal, D. A. (1990). Absorptive capacity: A new perspective on learning and innovation. Administrative Science Quarterly, 35(1), 128–152. https://doi.org/10.2307/2393553

Converse, J. M. (1987). Survey research in the United States: Roots and emergence, 1890–1960. University of California Press.

Cooper, A. (1999). The inmates are running the asylum. Sams.

Cooperrider, D. L., & Srivastva, S. (1987). Appreciative inquiry in organizational life. Research in Organizational Change and Development, 1, 129–169.

Cressey, D. R. (1953). Other people's money: A study in the social psychology of embezzlement. Free Press.

Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). SAGE.

Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555

Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993).

De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., Schroeder, T. V., Sox, H. C., & Van Der Weyden, M. B. (2004). Clinical trial registration: A statement from the International Committee of Medical Journal Editors. New England Journal of Medicine, 351(12), 1250–1251. https://doi.org/10.1056/NEJMe048225

de Leeuw, E. D. (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21(2), 233–255.

Denzin, N. K. (1978). The research act: A theoretical introduction to sociological methods (2nd ed.). McGraw-Hill.

Descartes, R. (1996). Meditations on first philosophy (J. Cottingham, Trans.; Rev. ed.). Cambridge University Press. (Original work published 1641)

Deville, J.-C., & Särndal, C.-E. (1992). Calibration estimators in survey sampling. Journal of the American Statistical Association, 87(418), 376–382. https://doi.org/10.1080/01621459.1992.10475217

Dewey, J. (1938). Logic: The theory of inquiry. Henry Holt.

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Wiley. https://doi.org/10.1002/9781394260645

Dixon, M., Freeman, K., & Toman, N. (2010). Stop trying to delight your customers. Harvard Business Review, 88(7/8), 116–122.

Dorminey, J., Fleming, A. S., Kranacher, M.-J., & Riley, R. A., Jr. (2012). The evolution of fraud theory. Issues in Accounting Education, 27(2), 555–579. https://doi.org/10.2308/iace-50131

Duncker, K. (1945). On problem-solving (L. S. Lees, Trans.). Psychological Monographs, 58(5), i–113. https://doi.org/10.1037/h0093599

Dwork, C., McSherry, F., Nissim, K., & Smith, A. (2006). Calibrating noise to sensitivity in private data analysis. In S. Halevi & T. Rabin (Eds.), Theory of cryptography (pp. 265–284). Springer. https://doi.org/10.1007/11681878_14

Edmondson, A. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44(2), 350–383. https://doi.org/10.2307/2666999

Edwards, W., Lindman, H., & Savage, L. J. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70(3), 193–242. https://doi.org/10.1037/h0044139

Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1–26. https://doi.org/10.1214/aos/1176344552

Eisenstein, E. L. (1979). The printing press as an agent of change. Cambridge University Press.

Elwert, F., & Winship, C. (2014). Endogenous selection bias: The problem of conditioning on a collider variable. Annual Review of Sociology, 40, 31–53. https://doi.org/10.1146/annurev-soc-071913-043455

Eppler, M. J., & Mengis, J. (2004). The concept of information overload: A review of literature from organization science, accounting, marketing, MIS, and related disciplines. The Information Society, 20(5), 325–344. https://doi.org/10.1080/01972240490507974

Etzkowitz, H., & Leydesdorff, L. (2000). The dynamics of innovation: From National Systems and "Mode 2" to a Triple Helix of university–industry–government relations. Research Policy, 29(2), 109–123. https://doi.org/10.1016/S0048-7333(99)00055-4

Facione, P. A. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction (The Delphi report). California Academic Press.

Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4(5), Article e5738. https://doi.org/10.1371/journal.pone.0005738

Financial Action Task Force. (2012). International standards on combating money laundering and the financing of terrorism & proliferation: The FATF recommendations. FATF/OECD.

Findley, K. A., & Scott, M. S. (2006). The multiple dimensions of tunnel vision in criminal cases. Wisconsin Law Review, 2006(2), 291–397.

Fisher, R. P., & Geiselman, R. E. (1992). Memory-enhancing techniques for investigative interviewing: The cognitive interview. Charles C Thomas.

Fishkin, J. S. (2009). When the people speak: Deliberative democracy and public consultation. Oxford University Press.

Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist, 34(10), 906–911. https://doi.org/10.1037/0003-066X.34.10.906

Floridi, L. (2005). Is semantic information meaningful data? Philosophy and Phenomenological Research, 70(2), 351–370. https://doi.org/10.1111/j.1933-1592.2005.tb00531.x

Foster, K. R., & Kokko, H. (2009). The evolution of superstitious and superstition-like behaviour. Proceedings of the Royal Society B: Biological Sciences, 276(1654), 31–37. https://doi.org/10.1098/rspb.2008.0981

Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484

Frické, M. (2009). The knowledge pyramid: A critique of the DIKW hierarchy. Journal of Information Science, 35(2), 131–142. https://doi.org/10.1177/0165551508094050

Fricker, M. (2007). Epistemic injustice: Power and the ethics of knowing. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780198237907.001.0001

Gadamer, H.-G. (2004). Truth and method (J. Weinsheimer & D. G. Marshall, Trans.; 2nd rev. ed.). Continuum. (Original work published 1960)

Galton, F. (1886). Regression towards mediocrity in hereditary stature. Journal of the Anthropological Institute of Great Britain and Ireland, 15, 246–263. https://doi.org/10.2307/2841583

Galton, F. (1907). Vox populi. Nature, 75(1949), 450–451. https://doi.org/10.1038/075450a0

Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2–3), 87–105. https://doi.org/10.1016/S1096-7516(00)00016-6

Geertz, C. (1973). The interpretation of cultures. Basic Books.

Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press. https://doi.org/10.1017/CBO9780511790942

Gettier, E. L. (1963). Is justified true belief knowledge? Analysis, 23(6), 121–123. https://doi.org/10.1093/analys/23.6.121

Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: The dynamics of science and research in contemporary societies. SAGE.

Gigerenzer, G., & Hoffrage, U. (1995). How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review, 102(4), 684–704. https://doi.org/10.1037/0033-295X.102.4.684

Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine.

Glassman, M., & Kang, M. J. (2012). Intelligence in the internet age: The emergence and evolution of Open Source Intelligence (OSINT). Computers in Human Behavior, 28(2), 673–682. https://doi.org/10.1016/j.chb.2011.11.014

Goldman, A. I. (1999). Knowledge in a social world. Oxford University Press. https://doi.org/10.1093/0198238207.001.0001

Graves, L. (2016). Deciding what's true: The rise of political fact-checking in American journalism. Columbia University Press.

Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. https://doi.org/10.1007/s10654-016-0149-3

Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998). Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology, 74(6), 1464–1480. https://doi.org/10.1037/0022-3514.74.6.1464

Griffin, A., & Hauser, J. R. (1993). The voice of the customer. Marketing Science, 12(1), 1–27. https://doi.org/10.1287/mksc.12.1.1

Groves, R. M., & Couper, M. P. (1998). Nonresponse in household interview surveys. Wiley. https://doi.org/10.1002/9781118490082

Groves, R. M., & Heeringa, S. G. (2006). Responsive design for household surveys: Tools for actively controlling survey errors and costs. Journal of the Royal Statistical Society: Series A (Statistics in Society), 169(3), 439–457. https://doi.org/10.1111/j.1467-985X.2006.00423.x

Groves, R. M., & Lyberg, L. (2010). Total survey error: Past, present, and future. Public Opinion Quarterly, 74(5), 849–879. https://doi.org/10.1093/poq/nfq065

Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72(2), 167–189. https://doi.org/10.1093/poq/nfn011

Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70(5), 646–675. https://doi.org/10.1093/poq/nfl033

Groves, R. M., Fowler, F. J., Jr., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology (2nd ed.). Wiley.

Groves, R. M., Presser, S., & Dipko, S. (2004). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68(1), 2–31. https://doi.org/10.1093/poq/nfh002

Groves, R. M., Singer, E., & Corning, A. (2000). Leverage-saliency theory of survey participation. Public Opinion Quarterly, 64(3), 299–308. https://doi.org/10.1086/317990

Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), 59–82. https://doi.org/10.1177/1525822X05279903

Gupta, S., Hanssens, D., Hardie, B., Kahn, W., Kumar, V., Lin, N., Ravishanker, N., & Sriram, S. (2006). Modeling customer lifetime value. Journal of Service Research, 9(2), 139–155. https://doi.org/10.1177/1094670506293810

Hamill, R., Wilson, T. D., & Nisbett, R. E. (1980). Insensitivity to sample bias: Generalizing from atypical cases. Journal of Personality and Social Psychology, 39(4), 578–589. https://doi.org/10.1037/0022-3514.39.4.578

Harding, S. (1986). The science question in feminism. Cornell University Press.

Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer. https://doi.org/10.1007/978-0-387-84858-7

Heckathorn, D. D. (1997). Respondent-driven sampling: A new approach to the study of hidden populations. Social Problems, 44(2), 174–199. https://doi.org/10.2307/3096941

Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004). A structural approach to selection bias. Epidemiology, 15(5), 615–625. https://doi.org/10.1097/01.ede.0000135174.63482.43

Heuer, R. J., Jr. (1999). Psychology of intelligence analysis. Center for the Study of Intelligence, Central Intelligence Agency.

Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235–266. https://doi.org/10.1023/B:EDPR.0000034022.16470.f3

Hogan, A., Blomqvist, E., Cochez, M., d'Amato, C., de Melo, G., Gutierrez, C., Kirrane, S., Labra Gayo, J. E., Navigli, R., Neumaier, S., Ngonga Ngomo, A.-C., Polleres, A., Rashid, S. M., Rula, A., Schmelzeisen, L., Sequeda, J., Staab, S., & Zimmermann, A. (2021). Knowledge graphs. ACM Computing Surveys, 54(4), Article 71. https://doi.org/10.1145/3447772

Hogan, C. E., Rezaee, Z., Riley, R. A., Jr., & Velury, U. K. (2008). Financial statement fraud: Insights from the academic literature. Auditing: A Journal of Practice & Theory, 27(2), 231–252. https://doi.org/10.2308/aud.2008.27.2.231

Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–960. https://doi.org/10.1080/01621459.1986.10478354

Horvitz, D. G., & Thompson, D. J. (1952). A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260), 663–685. https://doi.org/10.1080/01621459.1952.10483446

Howard, R. A. (1966). Information value theory. IEEE Transactions on Systems Science and Cybernetics, 2(1), 22–26. https://doi.org/10.1109/TSSC.1966.300074

Huber, G. P. (1991). Organizational learning: The contributing processes and the literatures. Organization Science, 2(1), 88–115. https://doi.org/10.1287/orsc.2.1.88

Hulnick, A. S. (2006). What's wrong with the intelligence cycle. Intelligence and National Security, 21(6), 959–979. https://doi.org/10.1080/02684520601046291

Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), Article e124. https://doi.org/10.1371/journal.pmed.0020124

Irwin, D., & Mandel, D. R. (2019). Improving information evaluation for intelligence production. Intelligence and National Security, 34(4), 503–525. https://doi.org/10.1080/02684527.2019.1569343

Janis, I. L. (1972). Victims of groupthink: A psychological study of foreign-policy decisions and fiascoes. Houghton Mifflin.

Jaynes, E. T. (2003). Probability theory: The logic of science (G. L. Bretthorst, Ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511790423

Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), Article 248. https://doi.org/10.1145/3571730

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953

Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80(4), 237–251. https://doi.org/10.1037/h0034747

Kahneman, D. (2011). Thinking, fast and slow. Farrar, Straus and Giroux.

Kahneman, D., Fredrickson, B. L., Schreiber, C. A., & Redelmeier, D. A. (1993). When more pain is preferred to less: Adding a better end. Psychological Science, 4(6), 401–405. https://doi.org/10.1111/j.1467-9280.1993.tb00589.x

Kalton, G., & Flores-Cervantes, I. (2003). Weighting methods. Journal of Official Statistics, 19(2), 81–98.

Kaplan, R. S., & Norton, D. P. (1992). The balanced scorecard: Measures that drive performance. Harvard Business Review, 70(1), 71–79.

Kapur, M. (2008). Productive failure. Cognition and Instruction, 26(3), 379–424. https://doi.org/10.1080/07370000802212669

Kassin, S. M., & Gudjonsson, G. H. (2004). The psychology of confessions: A review of the literature and issues. Psychological Science in the Public Interest, 5(2), 33–67. https://doi.org/10.1111/j.1529-1006.2004.00016.x

Kassin, S. M., Dror, I. E., & Kukucka, J. (2013). The forensic confirmation bias: Problems, perspectives, and proposed solutions. Journal of Applied Research in Memory and Cognition, 2(1), 42–52. https://doi.org/10.1016/j.jarmac.2013.01.001

Keiningham, T. L., Cooil, B., Andreassen, T. W., & Aksoy, L. (2007). A longitudinal examination of net promoter and firm revenue growth. Journal of Marketing, 71(3), 39–51. https://doi.org/10.1509/jmkg.71.3.039

Kent, S. (1949). Strategic intelligence for American world policy. Princeton University Press.

Kent, S. (1964). Words of estimative probability. Studies in Intelligence, 8(4), 49–65.

Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. https://doi.org/10.1207/s15327957pspr0203_4

King, G., Murray, C. J. L., Salomon, J. A., & Tandon, A. (2004). Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98(1), 191–207. https://doi.org/10.1017/S000305540400108X

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75–86. https://doi.org/10.1207/s15326985ep4102_1

Kish, L. (1965). Survey sampling. Wiley.

Kish, L. (1992). Weighting for unequal Pi. Journal of Official Statistics, 8(2), 183–200.

Kitchin, R. (2014). The data revolution: Big data, open data, data infrastructures and their consequences. SAGE. https://doi.org/10.4135/9781473909472

Knaflic, C. N. (2015). Storytelling with data: A data visualization guide for business professionals. Wiley.

Knight, F. H. (1921). Risk, uncertainty and profit. Houghton Mifflin.

Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 140–181. https://doi.org/10.1007/s10618-008-0114-1

Kounios, J., & Beeman, M. (2014). The cognitive neuroscience of insight. Annual Review of Psychology, 65, 71–93. https://doi.org/10.1146/annurev-psych-010213-115154

Kovach, B., & Rosenstiel, T. (2014). The elements of journalism: What newspeople should know and the public should expect (3rd ed.). Three Rivers Press.

Kozinets, R. V. (2002). The field behind the screen: Using netnography for marketing research in online communities. Journal of Marketing Research, 39(1), 61–72. https://doi.org/10.1509/jmkr.39.1.61.18935

Kreuter, F. (Ed.). (2013). Improving surveys with paradata: Analytic uses of process information. Wiley. https://doi.org/10.1002/9781118596869

Krippendorff, K. (2019). Content analysis: An introduction to its methodology (4th ed.). SAGE.

Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236. https://doi.org/10.1002/acp.2350050305

Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537–567. https://doi.org/10.1146/annurev.psych.50.1.537

Kuhn, T. S. (1962). The structure of scientific revolutions. University of Chicago Press.

LaPiere, R. T. (1934). Attitudes vs. actions. Social Forces, 13(2), 230–237. https://doi.org/10.2307/2570339

Leary, M. R., Diebels, K. J., Davisson, E. K., Jongman-Sereno, K. P., Isherwood, J. C., Raimi, K. T., Deffler, S. A., & Hoyle, R. H. (2017). Cognitive and interpersonal features of intellectual humility. Personality and Social Psychology Bulletin, 43(6), 793–813. https://doi.org/10.1177/0146167217697695

Lemon, K. N., & Verhoef, P. C. (2016). Understanding customer experience throughout the customer journey. Journal of Marketing, 80(6), 69–96. https://doi.org/10.1509/jm.15.0420

Lewandowsky, S., Ecker, U. K. H., Seifert, C. M., Schwarz, N., & Cook, J. (2012). Misinformation and its correction: Continued influence and successful debiasing. Psychological Science in the Public Interest, 13(3), 106–131. https://doi.org/10.1177/1529100612451018

Lewin, K. (1946). Action research and minority problems. Journal of Social Issues, 2(4), 34–46. https://doi.org/10.1111/j.1540-4560.1946.tb02295.x

Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 22(140), 1–55.

Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. SAGE.

Lipton, P. (2004). Inference to the best explanation (2nd ed.). Routledge.

Little, R. J. A., & Rubin, D. B. (2019). Statistical analysis with missing data (3rd ed.). Wiley. https://doi.org/10.1002/9781119482260

Loewenstein, G. (1994). The psychology of curiosity: A review and reinterpretation. Psychological Bulletin, 116(1), 75–98. https://doi.org/10.1037/0033-2909.116.1.75

Loftus, E. F., & Palmer, J. C. (1974). Reconstruction of automobile destruction: An example of the interaction between language and memory. Journal of Verbal Learning and Verbal Behavior, 13(5), 585–589. https://doi.org/10.1016/S0022-5371(74)80011-3

Lohr, S. L. (2021). Sampling: Design and analysis (3rd ed.). CRC Press. https://doi.org/10.1201/9780429298899

Luca, M., & Zervas, G. (2016). Fake it till you make it: Reputation, competition, and Yelp review fraud. Management Science, 62(12), 3412–3427. https://doi.org/10.1287/mnsc.2015.2304

Luhn, H. P. (1958). A business intelligence system. IBM Journal of Research and Development, 2(4), 314–319. https://doi.org/10.1147/rd.24.0314

Machlup, F. (1962). The production and distribution of knowledge in the United States. Princeton University Press.

Mandel, D. R., & Barnes, A. (2014). Accuracy of forecasts in strategic intelligence. Proceedings of the National Academy of Sciences, 111(30), 10984–10989. https://doi.org/10.1073/pnas.1406138111

Mangel, M., & Samaniego, F. J. (1984). Abraham Wald's work on aircraft survivability. Journal of the American Statistical Association, 79(386), 259–267. https://doi.org/10.1080/01621459.1984.10478038

McIntyre, L. (2018). Post-truth. MIT Press.

Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085

Meadows, D. H. (2008). Thinking in systems: A primer. Chelsea Green.

Meiklejohn, S., Pomarole, M., Jordan, G., Levchenko, K., McCoy, D., Voelker, G. M., & Savage, S. (2013). A fistful of bitcoins: Characterizing payments among men with no names. In Proceedings of the 2013 Conference on Internet Measurement (pp. 127–140). ACM. https://doi.org/10.1145/2504730.2504747

Mellers, B., Stone, E., Murray, T., Minster, A., Rohrbaugh, N., Bishop, M., Chen, E., Baker, J., Hou, Y., Horowitz, M., Ungar, L., & Tetlock, P. (2015). Identifying and cultivating superforecasters as a method of improving probabilistic predictions. Perspectives on Psychological Science, 10(3), 267–281. https://doi.org/10.1177/1745691615577794

Memon, A., Meissner, C. A., & Fraser, J. (2010). The cognitive interview: A meta-analytic review and study space analysis of the past 25 years. Psychology, Public Policy, and Law, 16(4), 340–372. https://doi.org/10.1037/a0020518

Merton, R. K., & Barber, E. (2004). The travels and adventures of serendipity: A study in sociological semantics and the sociology of science. Princeton University Press.

Merton, R. K. (1965). On the shoulders of giants: A Shandean postscript. Free Press.

Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations (N. W. Storer, Ed.). University of Chicago Press.

Messerli, F. H. (2012). Chocolate consumption, cognitive function, and Nobel laureates. New England Journal of Medicine, 367(16), 1562–1564. https://doi.org/10.1056/NEJMon1211064

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741

Metropolis, N., & Ulam, S. (1949). The Monte Carlo method. Journal of the American Statistical Association, 44(247), 335–341. https://doi.org/10.1080/01621459.1949.10483310

Mick, D. G. (1986). Consumer research and semiotics: Exploring the morphology of signs, symbols, and significance. Journal of Consumer Research, 13(2), 196–213. https://doi.org/10.1086/209060

Milne, R., & Bull, R. (1999). Investigative interviewing: Psychology and practice. Wiley.

Moreau, L., & Missier, P. (Eds.). (2013). PROV-DM: The PROV data model (W3C Recommendation). World Wide Web Consortium. https://www.w3.org/TR/prov-dm/

Moustakas, C. (1990). Heuristic research: Design, methodology, and applications. SAGE.

Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, Article 0021. https://doi.org/10.1038/s41562-016-0021

Murphy, A. H. (1973). A new vector partition of the probability score. Journal of Applied Meteorology, 12(4), 595–600. https://doi.org/10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2

National Research Council. (2009). Strengthening forensic science in the United States: A path forward. National Academies Press. https://doi.org/10.17226/12589

Near, J. P., & Miceli, M. P. (1985). Organizational dissidence: The case of whistle-blowing. Journal of Business Ethics, 4(1), 1–16. https://doi.org/10.1007/BF00382668

Nelder, J. A., & Wedderburn, R. W. M. (1972). Generalized linear models. Journal of the Royal Statistical Society: Series A (General), 135(3), 370–384. https://doi.org/10.2307/2344614

Neter, J., & Waksberg, J. (1964). A study of response errors in expenditures data from household interviews. Journal of the American Statistical Association, 59(305), 18–55. https://doi.org/10.1080/01621459.1964.10480699

Neyman, J. (1934). On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection. Journal of the Royal Statistical Society, 97(4), 558–625. https://doi.org/10.2307/2342192

Neyman, J. (1937). Outline of a theory of statistical estimation based on the classical theory of probability. Philosophical Transactions of the Royal Society of London. Series A, 236(767), 333–380. https://doi.org/10.1098/rsta.1937.0005

Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. https://doi.org/10.1037/1089-2680.2.2.175

Nonaka, I., & Takeuchi, H. (1995). The knowledge-creating company: How Japanese companies create the dynamics of innovation. Oxford University Press.

Nonaka, I. (1994). A dynamic theory of organizational knowledge creation. Organization Science, 5(1), 14–37. https://doi.org/10.1287/orsc.5.1.14

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114

Ohno, T. (1988). Toyota production system: Beyond large-scale production. Productivity Press.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716

Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872. https://doi.org/10.1016/j.jesp.2009.03.009

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, Article n71. https://doi.org/10.1136/bmj.n71

Pariser, E. (2011). The filter bubble: What the Internet is hiding from you. Penguin Press.

Park, D. K., Gelman, A., & Bafumi, J. (2004). Bayesian multilevel estimation with poststratification: State-level estimates from national polls. Political Analysis, 12(4), 375–385. https://doi.org/10.1093/pan/mph024

Paul, R., & Elder, L. (2006). The thinker's guide to the art of Socratic questioning. Foundation for Critical Thinking.

Pawson, R., & Tilley, N. (1997). Realistic evaluation. SAGE.

Pearl, J., & Mackenzie, D. (2018). The book of why: The new science of cause and effect. Basic Books.

Pearl, J. (2009). Causality: Models, reasoning, and inference (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511803161

Pearson, K. (1897). Mathematical contributions to the theory of evolution: On a form of spurious correlation which may arise when indices are used in the measurement of organs. Proceedings of the Royal Society of London, 60, 489–498. https://doi.org/10.1098/rspl.1896.0076

Pearson, K. (1900). On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 50(302), 157–175. https://doi.org/10.1080/14786440009463897

Pearson, K. (1901). On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11), 559–572. https://doi.org/10.1080/14786440109462720

Peirce, C. S. (1877). The fixation of belief. Popular Science Monthly, 12, 1–15.

Plassmann, H., Ramsøy, T. Z., & Milosavljevic, M. (2012). Branding the brain: A critical review and outlook. Journal of Consumer Psychology, 22(1), 18–36. https://doi.org/10.1016/j.jcps.2011.11.010

Polanyi, M. (1966). The tacit dimension. Doubleday.

Popper, K. R. (1959). The logic of scientific discovery. Hutchinson.

Porter, M. E. (1979). How competitive forces shape strategy. Harvard Business Review, 57(2), 137–145.

Powell, W. W., & Snellman, K. (2004). The knowledge economy. Annual Review of Sociology, 30, 199–220. https://doi.org/10.1146/annurev.soc.29.010202.100037

Ragin, C. C. (1987). The comparative method: Moving beyond qualitative and quantitative strategies. University of California Press.

Rao, J. N. K., & Molina, I. (2015). Small area estimation (2nd ed.). Wiley. https://doi.org/10.1002/9781118735855

Reason, J. (1990). Human error. Cambridge University Press. https://doi.org/10.1017/CBO9781139062367

Reason, J. (2000). Human error: Models and management. BMJ, 320(7237), 768–770. https://doi.org/10.1136/bmj.320.7237.768

Reichheld, F. F. (2003). The one number you need to grow. Harvard Business Review, 81(12), 46–54.

Ries, E. (2011). The lean startup. Crown Business.

Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4(2), 155–169. https://doi.org/10.1007/BF01405730

Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15(3), 351–357. https://doi.org/10.2307/2087176

Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press.

Roozenbeek, J., & van der Linden, S. (2019). Fake news game confers psychological resistance against online misinformation. Palgrave Communications, 5, Article 65. https://doi.org/10.1057/s41599-019-0279-9

Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638

Rothstein, D., & Santana, L. (2011). Make just one change: Teach students to ask their own questions. Harvard Education Press.

Rousseau, D. M. (2006). Is there such a thing as "evidence-based management"? Academy of Management Review, 31(2), 256–269. https://doi.org/10.5465/amr.2006.20208679

Rowley, J. (2007). The wisdom hierarchy: Representations of the DIKW hierarchy. Journal of Information Science, 33(2), 163–180. https://doi.org/10.1177/0165551506070706

Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701. https://doi.org/10.1037/h0037350

Ryle, G. (1949). The concept of mind. Hutchinson.

Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021). An excess of positive results: Comparing the standard psychology literature with Registered Reports. Advances in Methods and Practices in Psychological Science, 4(2). https://doi.org/10.1177/25152459211007467

Schoemaker, P. J. H. (1995). Scenario planning: A tool for strategic thinking. Sloan Management Review, 36(2), 25–40.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.

Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x

Shapin, S. (1996). The scientific revolution. University of Chicago Press.

Shapiro, C., & Varian, H. R. (1999). Information rules: A strategic guide to the network economy. Harvard Business School Press.

Sheeran, P. (2002). Intention–behavior relations: A conceptual and empirical review. European Review of Social Psychology, 12(1), 1–36. https://doi.org/10.1080/14792772143000003

Shewhart, W. A. (1931). Economic control of quality of manufactured product. Van Nostrand.

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632

Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series B (Methodological), 13(2), 238–241. https://doi.org/10.1111/j.2517-6161.1951.tb00088.x

Sio, U. N., & Ormerod, T. C. (2009). Does incubation enhance problem solving? A meta-analytic review. Psychological Bulletin, 135(1), 94–120. https://doi.org/10.1037/a0014212

Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178–182. https://doi.org/10.1177/014107680609900414

Smith, W. R. (1956). Product differentiation and market segmentation as alternative marketing strategies. Journal of Marketing, 21(1), 3–8. https://doi.org/10.1177/002224295602100102

Snow, J. (1855). On the mode of communication of cholera (2nd ed.). John Churchill.

Sorensen, R. A. (1992). Thought experiments. Oxford University Press.

Sparrow, M. K. (1991). The application of network analysis to criminal intelligence: An assessment of the prospects. Social Networks, 13(3), 251–274. https://doi.org/10.1016/0378-8733(91)90008-H

Spearman, C. (1904). "General intelligence," objectively determined and measured. The American Journal of Psychology, 15(2), 201–292. https://doi.org/10.2307/1412107

Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103(2684), 677–680. https://doi.org/10.1126/science.103.2684.677

Student. (1908). The probable error of a mean. Biometrika, 6(1), 1–25. https://doi.org/10.1093/biomet/6.1.1

Surowiecki, J. (2004). The wisdom of crowds. Doubleday.

Tetlock, P. E., & Gardner, D. (2015). Superforecasting: The art and science of prediction. Crown.

Thaler, R. H., & Sunstein, C. R. (2008). Nudge: Improving decisions about health, wealth, and happiness. Yale University Press.

Thistlethwaite, D. L., & Campbell, D. T. (1960). Regression-discontinuity analysis: An alternative to the ex post facto experiment. Journal of Educational Psychology, 51(6), 309–317. https://doi.org/10.1037/h0044319

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267–288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x

Tourangeau, R., & Yan, T. (2007). Sensitive questions in surveys. Psychological Bulletin, 133(5), 859–883. https://doi.org/10.1037/0033-2909.133.5.859

Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press. https://doi.org/10.1017/CBO9780511819322

Tufte, E. R. (1983). The visual display of quantitative information. Graphics Press.

Tukey, J. W. (1977). Exploratory data analysis. Addison-Wesley.

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131. https://doi.org/10.1126/science.185.4157.1124

Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211(4481), 453–458. https://doi.org/10.1126/science.7455683

Twardy, C. R. (2004). Argument maps improve critical thinking. Teaching Philosophy, 27(2), 95–116. https://doi.org/10.5840/teachphil200427213

Vosoughi, S., Roy, D., & Aral, S. (2018). The spread of true and false news online. Science, 359(6380), 1146–1151. https://doi.org/10.1126/science.aap9559

Vrij, A. (2008). Detecting lies and deceit: Pitfalls and opportunities (2nd ed.). Wiley.

Wallas, G. (1926). The art of thought. Harcourt, Brace.

Walsh, J. P., & Ungson, G. R. (1991). Organizational memory. Academy of Management Review, 16(1), 57–91. https://doi.org/10.5465/amr.1991.4278992

Walton, D. (2008). Informal logic: A pragmatic approach (2nd ed.). Cambridge University Press.

Wardle, C., & Derakhshan, H. (2017). Information disorder: Toward an interdisciplinary framework for research and policy making (Council of Europe report DGI(2017)09). Council of Europe.

Warner, S. L. (1965). Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association, 60(309), 63–69. https://doi.org/10.1080/01621459.1965.10480775

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108

Weick, K. E. (1995). Sensemaking in organizations. SAGE.

Wells, G. L., & Olson, E. A. (2003). Eyewitness testimony. Annual Review of Psychology, 54, 277–295. https://doi.org/10.1146/annurev.psych.54.101601.145028

Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge University Press. https://doi.org/10.1017/CBO9780511803932

Wilensky, H. L. (1967). Organizational intelligence: Knowledge and policy in government and industry. Basic Books.

Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., … Mons, B. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, Article 160018. https://doi.org/10.1038/sdata.2016.18

Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. SAGE. https://doi.org/10.4135/9781412983655

Wilson, A. M. (2001). Mystery shopping: Using deception to measure service performance. Psychology & Marketing, 18(7), 721–734. https://doi.org/10.1002/mar.1027

Wolfe, D. T., & Hermanson, D. R. (2004). The fraud diamond: Considering the four elements of fraud. The CPA Journal, 74(12), 38–42.

Wolfers, J., & Zitzewitz, E. (2004). Prediction markets. Journal of Economic Perspectives, 18(2), 107–126. https://doi.org/10.1257/0895330041371321

Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., & Malone, T. W. (2010). Evidence for a collective intelligence factor in the performance of human groups. Science, 330(6004), 686–688. https://doi.org/10.1126/science.1193147

Yin, R. K. (2018). Case study research and applications: Design and methods (6th ed.). SAGE.

Yule, G. U. (1926). Why do we sometimes get nonsense-correlations between time-series? A study in sampling and the nature of time-series. Journal of the Royal Statistical Society, 89(1), 1–63. https://doi.org/10.2307/2341482

Zenko, M. (2015). Red team: How to succeed by thinking like the enemy. Basic Books.

Zins, C. (2007). Conceptual approaches for defining data, information, and knowledge. Journal of the American Society for Information Science and Technology, 58(4), 479–493. https://doi.org/10.1002/asi.20508