Glossary
ModQu glossary
Definitions of the key terms used across ModQu's articles on surveys, research, statistics, intelligence, insights, inquiry, investigation and the generation of knowledge, with sources.
- Surveys
Surveys glossaryTerms used in designing, sampling, fielding and checking surveys, from questionnaire wording to weighting and data quality.
40 terms · 37 references - Research
Research glossaryTerms for research philosophy, design, qualitative and quantitative methods, evidence synthesis and research integrity.
46 terms · 42 references - Statistics
Statistics glossaryTerms for describing data, drawing inferences, building models and reasoning about cause, with the common ways statistics mislead.
62 terms · 57 references - Intelligence
Intelligence glossaryTerms from intelligence analysis and business, market, competitive and open-source intelligence, including forecasting and analytic tradecraft.
36 terms · 33 references - Insights
Insights glossaryTerms for what an insight is, how insights are generated from customer and market evidence, and how they go wrong.
30 terms · 30 references - Inquiry
Inquiry glossaryTerms for the philosophy and practice of asking questions, reasoning, learning through inquiry and inquiry in organisations and public life.
42 terms · 37 references - Investigation
Investigation glossaryTerms for evidence, interviewing, fraud, digital and scientific investigation, and the biases investigators must guard against.
31 terms · 31 references - The Generation of Knowledge
The Generation of Knowledge glossaryTerms for what knowledge is, how it is produced, managed and valued, and how it is trusted, preserved and shared.
41 terms · 46 references
All terms A–Z
A
- A/B test also: split test
A randomised controlled experiment, usually online, in which users are randomly assigned to two or more versions of a product, page or message and an outcome such as conversion is compared. Randomisation makes the versions comparable, so differences in outcome can be attributed to the change. Common pitfalls include stopping a test as soon as results look significant and testing many metrics without correction. (Kohavi et al., 2009)
- Abduction also: inference to the best explanation
Reasoning to the hypothesis that would best explain a surprising observation. Peirce treated it as the only form of inference that introduces new ideas; contemporary philosophy discusses it as inference to the best explanation, in which explanations are compared on criteria such as scope, simplicity and coherence with what is already known. (Lipton, 2004)
- Absorptive capacity
A firm's ability to recognise the value of new external information, assimilate it and apply it to commercial ends. Cohen and Levinthal argued that it depends largely on prior related knowledge, which is why firms that do their own research are better at using the research of others. (Cohen & Levinthal, 1990)
- Acquiescence bias also: yea-saying
The tendency of some respondents to agree with statements regardless of their content, sometimes called "yea-saying". It inflates agreement on agree–disagree items and can create spurious correlations between items that share the same direction of wording. Common remedies are item-specific question formats (asking directly about the attribute rather than asking for agreement) and balanced scales that mix positively and negatively worded items. (Krosnick, 1999; Tourangeau et al., 2000)
- Action research
An approach in which researchers and participants study a practical problem while acting to change it, moving through repeated cycles of planning, action, observation and reflection. Kurt Lewin coined the term in the 1940s. It values practical improvement and the knowledge of participants alongside formal findings. (Lewin, 1946)
- Actionable insight
An insight specific and credible enough to change a decision: it identifies who is affected, what to do differently and why, and it is backed by evidence that decision-makers trust. Findings that are interesting but do not point to an action, or that the organisation cannot act on, fall short of this standard. (Rousseau, 2006)
- Admissibility
Whether evidence may be considered by a court or tribunal. Rules vary by jurisdiction but usually require relevance, reliability and lawful collection. For expert scientific testimony in US federal courts, the Daubert standard asks whether a method has been tested, peer reviewed, has a known error rate and is generally accepted. (Daubert v. Merrell Dow Pharmaceuticals, Inc., 1993; National Research Council, 2009)
- Aha moment also: eureka moment
The sudden, subjectively surprising realisation of a solution or understanding that accompanies insight problem solving. Neuroscience research links it to a burst of activity in the right anterior temporal lobe shortly before the solution reaches awareness, preceded by a brief reduction in visual input as attention turns inward. (Kounios & Beeman, 2014)
- All-source intelligence also: all-source fusion
Intelligence produced by combining information from every available collection discipline, such as human sources, signals, imagery and open sources, into one assessment. Fusing sources allows analysts to corroborate reports and compensate for the gaps and biases of any single source. (Kent, 1949)
- Analysis of competing hypotheses also: ACH
A structured analytic technique developed by Richards Heuer in which analysts list all plausible hypotheses, array the evidence against each in a matrix, and focus on disconfirming evidence: the most likely hypothesis is the one with the least evidence against it, not the most evidence for it. It is designed to counter confirmation bias and premature closure. (Heuer, 1999)
- Anchoring vignettes
Short descriptions of hypothetical people or situations that every respondent rates on the same scale used for their self-assessment. Because the vignettes are fixed, differences in how groups rate them reveal differences in how those groups use the scale, which can then be corrected. The method addresses differential item functioning, the problem that two people with the same true level may choose different answer categories. (King et al., 2004)
- Anti-money laundering also: AML
Laws, controls and investigations designed to stop criminals from disguising the proceeds of crime as legitimate funds. Laundering is usually described in three stages: placement, layering and integration. The Financial Action Task Force recommendations set international standards, including customer due diligence and suspicious transaction reporting. (Financial Action Task Force, 2012)
- Apophenia
The tendency to perceive meaningful patterns or connections in random or unrelated data. It may have evolved because the cost of missing a real pattern often exceeded the cost of seeing a false one. In data analysis, large datasets with many variables make spurious patterns easy to find and easy to mistake for insight. (Foster & Kokko, 2009)
- Appreciative inquiry
An approach to organisational change, developed by David Cooperrider and Suresh Srivastva, that asks what works well and builds on it rather than diagnosing problems. It typically moves through discovery, dream, design and destiny phases. (Cooperrider & Srivastva, 1987)
- Argument mapping
Drawing the structure of an argument as a diagram that shows the main claim, the reasons for and objections against it, and how they relate. Studies in university courses found that practice with argument maps improved students' critical-thinking test scores more than conventional teaching. (Twardy, 2004)
- Attention check also: instructed response item
An item placed in a questionnaire to detect respondents who are not reading the questions, for example an instruction to select a specific answer. Failure rates are used to flag or remove inattentive cases. Evidence suggests such checks identify careless responding but can also irritate attentive respondents, so they are best combined with other indicators such as completion time and response patterns. (Oppenheimer et al., 2009; Meade & Craig, 2012)
B
- Base rate neglect also: base rate fallacy
The tendency to ignore the prior frequency of an event when judging the probability of a case, relying instead on specific information such as a test result. It leads people to overestimate the chance that a positive test means disease when the disease is rare. Presenting information as natural frequencies (for example, 9 out of 1,000) greatly reduces the error. (Bar-Hillel, 1980; Gigerenzer & Hoffrage, 1995)
- Bayes' theorem
A rule for updating the probability of a hypothesis in light of evidence: the posterior probability is proportional to the prior probability times the likelihood of the evidence under the hypothesis. Formally, P(H | E) = P(E | H) × P(H) / P(E). It underlies Bayesian statistics and much of rational belief revision. (Bayes, 1763; Jaynes, 2003)
- Bayesian inference
An approach to statistics in which unknown quantities are treated as having probability distributions. A prior distribution expressing what is known before the data is combined with the likelihood of the data to give a posterior distribution, from which estimates and credible intervals are read directly as probability statements about the parameters. (Edwards et al., 1963; Jaynes, 2003)
- Behavioural economics
The study of how psychological factors such as limited attention, loss aversion, present bias and social norms make economic choices depart from the predictions of standard rational-choice models. It supplies many consumer insights, for example that defaults strongly shape what people choose. (Tversky & Kahneman, 1974; Kahneman, 2011; Thaler & Sunstein, 2008)
- Blockchain analysis
Tracing cryptocurrency transactions on public ledgers to link addresses to each other and to real-world entities such as exchanges. Although users are pseudonymous, studies of Bitcoin have shown that clustering heuristics can follow flows of funds through many transactions. (Meiklejohn et al., 2013)
- Bloom's taxonomy
A classification of learning objectives by cognitive level. The 2001 revision orders them as remember, understand, apply, analyse, evaluate and create, and is often used to design questions that move from recall to higher-order thinking. (Anderson & Krathwohl, 2001)
- Bootstrap
A resampling method introduced by Bradley Efron in 1979 that estimates the sampling distribution of a statistic by repeatedly drawing samples, with replacement, from the observed data and recomputing the statistic each time. The spread of the resampled values gives standard errors and confidence intervals without strong distributional assumptions. (Efron, 1979)
- Brier score
A measure of the accuracy of probability forecasts, equal to the mean squared difference between the forecast probabilities and the outcomes (1 if the event happened, 0 if not). Lower is better. It can be decomposed into calibration and resolution, and it was the main scoring rule in the forecasting tournaments that identified superforecasters. (Brier, 1950; Murphy, 1973)
- Business intelligence also: BI
The processes, technologies and practices an organisation uses to collect, integrate, analyse and present data about its operations and environment in support of decisions. Hans Peter Luhn used the term in 1958 for an automated system to disseminate information; it now usually refers to data warehouses, reporting and dashboards. (Luhn, 1958; Chen et al., 2012)
C
- Calibration
The match between stated probabilities and observed frequencies: a forecaster is well calibrated if events given a 70% chance happen about 70% of the time. Calibration can be assessed with calibration curves and the calibration component of the Brier score, and it improves with training and feedback. (Murphy, 1973; Mandel & Barnes, 2014)
- Careless responding also: insufficient effort responding
Answering survey items without adequate attention to their content, producing responses that do not reflect the respondent's true state. Indicators include very fast completion, long strings of identical answers, inconsistent answers to items that should agree, and failed attention checks. Careless responding adds noise and can bias estimates and correlations if it is not detected. (Meade & Craig, 2012)
- Case study
An in-depth empirical study of a contemporary phenomenon in its real-life context, especially when the boundary between phenomenon and context is unclear. Case studies use multiple sources of evidence and can be single or multiple, exploratory, descriptive or explanatory. They generalise analytically, to theory, rather than statistically, to a population. (Yin, 2018)
- Central limit theorem
The result that the distribution of the mean (or sum) of many independent observations with finite variance approaches a normal distribution as the sample size grows, whatever the shape of the original distribution. It explains why the normal distribution is used so widely for sampling distributions and confidence intervals. (Lohr, 2021)
- Chain of custody
The documented, unbroken record of who collected, handled, transferred, stored and analysed an item of evidence, and when. It shows that the evidence presented is the same as that collected and has not been altered. In digital investigations it is supported by hash values that show files have not changed. (Casey, 2011; National Research Council, 2009)
- Chi-square test
A family of tests introduced by Karl Pearson in 1900 that compare observed frequencies with those expected under a hypothesis. The test of independence asks whether two categorical variables in a cross-tabulation are associated. With complex survey samples, design-adjusted versions such as the Rao–Scott correction are needed. (Pearson, 1900)
- Churn also: attrition
The loss of customers over a period, often expressed as a rate. Churn insight combines behavioural data showing who leaves and when with survey and interview data explaining why, so that retention efforts can target the causes rather than the symptoms. (Gupta et al., 2006)
- Citizen science
Scientific research in which members of the public take part in collecting, analysing or interpreting data, for example in bird counts. It can greatly extend the scale of data collection and public understanding of science, with data quality managed through training and validation. (Bonney et al., 2009)
- Cluster analysis
Methods that group cases so that cases in the same cluster are more similar to each other than to those in other clusters, using distances such as Euclidean distance. K-means and hierarchical clustering are common. In market research it is used to build segments, whose usefulness depends on the variables chosen. (Hastie et al., 2009)
- Cluster sampling
A probability design in which the population is divided into groups (clusters), such as schools or city blocks, a sample of clusters is selected, and units are surveyed within the selected clusters. It reduces travel and listing costs but usually lowers precision because units in the same cluster tend to be alike, which raises the design effect. (Kish, 1965; Lohr, 2021)
- Coding
Labelling segments of qualitative data, such as interview passages, with short descriptive or interpretive codes so that they can be compared, grouped into categories and developed into themes or theory. Codes may come from the data (inductive) or from an existing framework (deductive). (Braun & Clarke, 2006; Glaser & Strauss, 1967)
- Cognitive interview
An interviewing method for witnesses, developed by Ronald Fisher and Edward Geiselman, based on principles of memory: rapport, mental reinstatement of the context, reporting everything, and recalling events in different orders and from different perspectives. A meta-analysis found that it elicits substantially more correct detail than standard interviews, with a small increase in errors. (Fisher & Geiselman, 1992; Memon et al., 2010)
- Cognitive interviewing
A pretesting method in which respondents answer draft survey questions while thinking aloud or responding to probes about how they understood each question and arrived at their answer. It reveals problems of comprehension, recall, judgement and response mapping before a survey is fielded. It is qualitative and usually involves small purposive samples. (Willis, 2005; Tourangeau et al., 2000)
- Cohen's kappa
A measure of agreement between two raters assigning cases to categories, corrected for the agreement expected by chance. Kappa is 1 for perfect agreement and 0 for chance-level agreement. It is widely used to check the reliability of coding in content analysis and qualitative research. (Cohen, 1960)
- Coherentism
The view that a belief is justified by its coherence with a wider system of beliefs, rather than by resting on basic foundations. Laurence BonJour developed an influential version, later revising his position. (BonJour, 1985)
- Collective intelligence
The capacity of a group to perform well across a range of tasks, beyond the abilities of its individual members. Research has found a general collective intelligence factor in small groups that is more strongly related to members' social sensitivity and equality of conversational turn-taking than to the average or maximum intelligence of members. (Woolley et al., 2010)
- Collider bias
Bias that arises when an analysis conditions on (selects on, stratifies by or adjusts for) a variable that is caused by both the exposure and the outcome. Conditioning on such a collider creates a spurious association between its causes, as in Berkson's finding that two diseases appear related in hospital samples. (Elwert & Winship, 2014; Berkson, 1946; Hernán et al., 2004)
- Communities of practice
Groups of people who share a concern or a craft and learn how to do it better by interacting regularly. Etienne Wenger described them through mutual engagement, joint enterprise and a shared repertoire, and showed that much organisational knowledge lives and develops in such communities rather than in formal structures. (Wenger, 1998)
- Community of inquiry
A group that pursues understanding together through shared questioning and reasoning. Peirce saw the community of investigators, rather than the individual, as the agent of scientific inquiry. In education the term names a framework of cognitive, social and teaching presence for collaborative online learning. (Peirce, 1877; Garrison et al., 1999)
- Competitive intelligence also: CI
The legal and ethical collection and analysis of information about competitors, markets and the business environment to support strategic and tactical decisions. It differs from industrial espionage in relying on public sources, interviews and observation rather than deception or theft. (Porter, 1979)
- Confidence interval
A range computed from sample data by a procedure that, over repeated samples, contains the true parameter a stated proportion of the time, such as 95%. A particular interval either contains the parameter or does not; the 95% describes the procedure, not the probability for one interval. Jerzy Neyman set out the theory in 1937. (Neyman, 1937; Greenland et al., 2016)
- Confirmation bias
The tendency to search for, interpret, favour and remember information in ways that support existing beliefs, and to give less weight to contrary evidence. It is among the most pervasive reasoning errors and affects analysts, investigators and researchers as much as lay people. (Nickerson, 1998)
- Confirmation bias in investigation
The tendency of investigators and forensic examiners to seek, interpret and weigh evidence in ways that confirm an initial suspect or theory. Contextual information, such as knowing a suspect has confessed, can influence even technical judgements like fingerprint comparisons. Blind procedures and sequential unmasking reduce it. (Kassin et al., 2013; Nickerson, 1998)
- Confounding
Distortion of the estimated effect of an exposure on an outcome because a third variable causes both. Confounding can create, hide or reverse an association. It is controlled by randomisation, or in observational data by adjustment, matching or stratification on the confounders, which requires that they are known and measured. (Pearl, 2009; Hernán et al., 2004)
- Construct validity
The degree to which a measure actually captures the theoretical concept (construct) it is meant to measure. It is established by showing that scores relate to other measures as theory predicts: strongly with measures of the same construct (convergent evidence) and weakly with measures of different constructs (discriminant evidence). Modern validity theory treats it as the unifying form of validity. (Cronbach & Meehl, 1955; Messick, 1995)
- Content analysis
A method for making replicable and valid inferences from texts, images or other communication by coding their content into categories according to explicit rules. Quantitative content analysis counts coded features; qualitative variants interpret them. Reliability is checked by having independent coders apply the same scheme. (Krippendorff, 2019)
- Continued influence effect
The tendency of misinformation to keep affecting reasoning after it has been clearly corrected, especially when the correction leaves a gap in the explanation that the false information filled. Corrections work best when they offer an alternative account. (Lewandowsky et al., 2012)
- Correlation coefficient
A number between −1 and +1 summarising the strength and direction of association between two variables. Pearson's r measures linear association; Spearman's rho measures monotonic association using ranks. A correlation does not show causation and can be distorted by outliers and by non-linear relationships, as Anscombe's quartet illustrates. (Anscombe, 1973)
- Corroboration
Independent evidence that confirms a claim or another piece of evidence. Corroboration is strong only when the sources are genuinely independent, so that one cannot be an echo of the other, and when the confirming detail could not easily have been known or guessed. (Heuer, 1999)
- Counterfactual
The outcome that would have occurred under a condition other than the one that actually happened. In the potential outcomes framework each unit has an outcome under treatment and under control, and the causal effect is their difference. Because only one is ever observed, causal inference is a problem of estimating the missing counterfactual. (Rubin, 1974; Holland, 1986)
- Counterintelligence
Activities to identify, deter and counter efforts by others to obtain an organisation's protected information or to deceive it. In business it covers protecting trade secrets and strategic plans from competitors, insiders and foreign intelligence services. (Kent, 1949)
- Coverage error
Error that arises when the sampling frame does not match the target population, because some members are missing from the frame (undercoverage) or the frame includes units outside the population (overcoverage). Bias results when those missing differ from those covered on the variables being measured. It is one of the error sources in the total survey error framework. (Groves et al., 2009; Lohr, 2021)
- Critical realism
A philosophy of science holding that reality exists independently of our knowledge of it, but that our knowledge is fallible and socially produced. It distinguishes the real (structures and mechanisms), the actual (events) and the empirical (what is observed), and directs research to the mechanisms that produce outcomes in particular contexts. Realist evaluation draws on it. (Pawson & Tilley, 1997)
- Critical thinking
Purposeful, self-regulatory judgement that results in interpretation, analysis, evaluation and inference, together with explanation of the evidence and reasoning behind it, according to the consensus definition produced by a panel of 46 experts in 1990. It also involves dispositions such as open-mindedness and inquisitiveness. (Facione, 1990)
- Cronbach's alpha
A coefficient of internal consistency for a multi-item scale, based on the number of items and the average inter-item covariance. Values closer to 1 indicate that items vary together. A high alpha does not prove that a scale measures a single construct, and alpha rises with the number of items. (Cronbach, 1951)
- Cross-validation
A method for estimating how well a model will predict new data by repeatedly fitting it on part of the data and testing it on the held-out remainder. In k-fold cross-validation the data are split into k parts and each part serves once as the test set. It is used to choose model complexity and avoid overfitting. (Hastie et al., 2009)
- Cumulative knowledge
Knowledge that builds on earlier work, so that each contribution extends, corrects or combines what came before. Robert Merton traced the aphorism "standing on the shoulders of giants", used by Newton, back to Bernard of Chartres in the twelfth century. (Merton, 1965)
- Curiosity
The desire to know or learn. George Loewenstein's information-gap theory proposes that curiosity arises when attention is drawn to a gap between what one knows and what one wants to know, and that it rises as the gap narrows, which explains why a little knowledge often makes people more curious. (Loewenstein, 1994)
- Customer effort score also: CES
A metric based on asking customers how much effort they had to make to get an issue resolved or a task done. Its proponents found that reducing customer effort predicted loyalty better than delighting customers. Like other single-question metrics, it is best interpreted alongside open-ended feedback. (Dixon et al., 2010)
- Customer experience also: CX
A customer's cognitive, emotional, sensory, social and behavioural responses to a company across every touchpoint of the customer journey, before, during and after purchase. Customer experience research maps these touchpoints and identifies the moments that most affect satisfaction and loyalty. (Lemon & Verhoef, 2016)
- Customer journey
The sequence of stages and touchpoints a customer passes through in becoming aware of, considering, buying, using and recommending or leaving a product or brand. Journey maps combine research evidence on each step to show where customers struggle or drop out. (Lemon & Verhoef, 2016)
- Customer lifetime value also: CLV
The present value of all future profits expected from a customer relationship, usually modelled from retention probability, purchase frequency, margin and a discount rate. It shifts attention from single transactions to the long-term worth of customers and guides how much to spend acquiring and retaining them. (Gupta et al., 2006)
D
- Dashboard
A visual display of the most important information needed to achieve one or more objectives, arranged on a single screen so it can be monitored at a glance. Good dashboards show a small number of decision-relevant indicators with context, such as targets and trends, rather than decoration. (Tufte, 1983; Chen et al., 2012)
- Data
Recorded observations, measurements or symbols, such as survey responses or transaction records, that have not yet been organised or interpreted for a purpose. Data are never fully raw: what is recorded depends on choices about what to measure and how. (Kitchin, 2014; Zins, 2007)
- Deception also: denial and deception
Deliberate efforts by an adversary to mislead an analyst or decision-maker by concealing the truth (denial) or presenting falsehoods (deception), so that the target acts against its own interests. Analysts counter it by checking source access, looking for anomalies and testing whether evidence could have been planted. (Heuer, 1999)
- Decision framing
Defining the decision to be made, its scope, objectives, alternatives and constraints before seeking answers. Research on framing effects shows that how a choice is described changes what people choose, so getting the frame right is part of good inquiry. (Tversky & Kahneman, 1981)
- Decision-making under uncertainty
Choosing between options when outcomes are not known. When probabilities can be estimated (risk), expected value and expected utility guide choice; when they cannot (Knightian uncertainty), approaches such as scenario planning, robust strategies and gathering more information become important. (Knight, 1921; Tversky & Kahneman, 1981)
- Deduction also: deductive reasoning
Reasoning in which the conclusion follows necessarily from the premises: if the premises are true and the argument is valid, the conclusion must be true. Deduction preserves truth but does not add information beyond what the premises already contain. (Walton, 2008)
- Deliberative inquiry also: citizens' assembly
Inquiry in which a group of citizens, often randomly selected, learns about an issue from balanced information and experts, discusses it and reaches considered judgements. Deliberative polls and citizens' assemblies show that informed opinion can differ markedly from initial opinion. (Fishkin, 2009)
- Descriptive, predictive and prescriptive analytics
A common classification of analytics by the question asked: descriptive (what happened), diagnostic (why it happened), predictive (what is likely to happen) and prescriptive (what should be done). Each stage requires more modelling and stronger assumptions than the one before. (Chen et al., 2012)
- Design effect also: deff
The ratio of the variance of an estimate under a complex sample design to the variance it would have under a simple random sample of the same size. Clustering and unequal weights usually raise it above 1; effective stratification can lower it. Dividing the actual sample size by the design effect gives the effective sample size, the number of simple random cases with the same precision. (Kish, 1965; Kish, 1992)
- Design thinking
A human-centred approach to innovation that uses empathy with users, problem framing, idea generation, prototyping and testing in iterative cycles. Its inquiry stance treats early solutions as questions to be tested with users rather than answers to be defended. (Brown, 2008)
- Difference-in-differences also: DiD
A method that estimates a causal effect by comparing the change in an outcome over time in a group exposed to a policy or event with the change in an unexposed group. It removes stable differences between the groups and common trends, under the key assumption that the groups would have followed parallel trends without the intervention. (Card & Krueger, 1994; Angrist & Pischke, 2009)
- Differential privacy
A mathematical definition of privacy for statistical releases: an algorithm is differentially private if its output distribution changes very little whether or not any one person's data are included. It is usually achieved by adding calibrated random noise, and it gives a measurable privacy budget, epsilon. The US Census Bureau used it for 2020 census products. (Dwork et al., 2006)
- Diffusion of innovations
Everett Rogers's theory of how new ideas and products spread through a social system over time, through adopter groups described as innovators, early adopters, early majority, late majority and laggards. Adoption depends on perceived advantage, compatibility, complexity, trialability and observability. (Rogers, 2003)
- Digital forensics
The identification, preservation, collection, examination, analysis and presentation of digital evidence from computers, phones, networks and online services, using methods that maintain integrity and can withstand legal scrutiny. Work is done on verified copies so the original is not altered. (Casey, 2011)
- DIKW pyramid also: knowledge hierarchy
A model that arranges data, information, knowledge and wisdom as a hierarchy in which each level is derived from the one below. Russell Ackoff's 1989 article made it widely known. Critics argue that the levels are poorly defined and that knowledge is needed to produce data in the first place, so the relationships run in both directions. (Ackoff, 1989; Rowley, 2007; Frické, 2009)
- Directed acyclic graph also: DAG
A diagram of variables (nodes) connected by arrows representing direct causal effects, with no path that returns to its starting point. DAGs make causal assumptions explicit and show which variables must be adjusted for to estimate an effect (the back-door criterion) and which must not, such as colliders and mediators. (Pearl, 2009; Pearl & Mackenzie, 2018)
- Disinformation
False information deliberately created and spread to deceive or cause harm. It is distinguished from misinformation, false information shared without intent to deceive, and malinformation, genuine information shared to cause harm. False news has been found to spread farther and faster online than true news. (Wardle & Derakhshan, 2017; Vosoughi et al., 2018)
- Double-loop learning
Chris Argyris and Donald Schön's term for learning that questions and changes the underlying goals, norms and assumptions behind actions, as opposed to single-loop learning, which corrects actions within existing assumptions. A thermostat that adjusts heating is single-loop; questioning the temperature setting is double-loop. (Argyris & Schön, 1978)
- Due diligence
A systematic investigation of a company, person or transaction before a decision such as an acquisition, investment, partnership or hire. It covers financial, legal, operational and reputational matters and aims to confirm what is claimed and uncover what is not disclosed. (Financial Action Task Force, 2012)
E
- Ecological fallacy
Inferring relationships about individuals from data on groups. William Robinson showed in 1950 that the correlation between literacy and foreign birth across US states had the opposite sign to the correlation among individuals. Group-level correlations can be strong even when individual-level ones are weak or reversed. (Robinson, 1950)
- Effect size
A quantitative measure of the magnitude of a difference or relationship, such as a mean difference, Cohen's d, an odds ratio or a correlation. Unlike a p-value it does not depend on sample size, so it indicates practical importance. Jacob Cohen's conventions of small, medium and large effects are widely used but were meant only as a fallback. (Cohen, 1988)
- Effective sample size
The size of a simple random sample that would give the same precision as the actual sample, once clustering and weighting are taken into account. For weighting alone, Kish's approximation is the square of the sum of weights divided by the sum of squared weights. Reporting it shows how much precision a design or a weighting scheme has cost. (Kish, 1965; Kish, 1992)
- Epidemiological investigation
Investigation of the occurrence, distribution and causes of disease in a population, including outbreak investigations that identify a source and route of transmission. John Snow's 1854 study of cholera around the Broad Street pump in London, which linked cases to one water source, is a founding example. (Snow, 1855)
- Epistemic injustice
Miranda Fricker's term for wrongs done to people in their capacity as knowers. Testimonial injustice occurs when prejudice leads a hearer to give a speaker less credibility; hermeneutical injustice occurs when a group lacks shared concepts to make sense of its experience. (Fricker, 2007)
- Ethnography
The study of a group's culture and everyday practices through extended participant observation, interviews and documents, aiming to understand the world from members' perspectives. Clifford Geertz described its goal as "thick description": interpreting behaviour within the web of meanings that gives it significance. (Geertz, 1973)
- Evidence
Information that tends to make a claim more or less probable. Investigators distinguish direct from circumstantial evidence, and testimonial, documentary, physical and digital evidence, and they assess each item for relevance, reliability and weight. A lead is a direction for inquiry; it becomes evidence only once verified. (National Research Council, 2009)
- Evidence-based management
Making managerial decisions by combining the best available research evidence with organisational data, professional expertise and stakeholder values, and evaluating the results. Denise Rousseau's 2006 presidential address to the Academy of Management helped establish the idea. (Rousseau, 2006)
- Expert witness
A person whose specialised knowledge allows them to give opinion evidence to a court on matters beyond ordinary experience, such as forensic accounting or DNA analysis. Their duty is to the court rather than to the party that retains them, and their methods may be tested for reliability. (Daubert v. Merrell Dow Pharmaceuticals, Inc., 1993)
- Explicit knowledge
Knowledge that can be articulated, codified and shared in words, numbers, documents, databases or procedures. It is easier to store and transfer than tacit knowledge, but it depends on tacit understanding to be interpreted and used. (Nonaka, 1994)
- External validity
The extent to which a causal relationship found in a study holds across other people, settings, treatments and outcomes. Laboratory experiments can be strong on internal validity but weak on external validity; field experiments and replication across contexts strengthen it. (Shadish et al., 2002)
- Eyewitness testimony
Accounts of events given by people who saw them. Research shows that memory is reconstructive and can be altered by leading questions, post-event information and stress; for example, the verb used in a question changed witnesses' estimates of car speed. Mistaken eyewitness identification is a leading factor in wrongful convictions. (Loftus & Palmer, 1974; Wells & Olson, 2003)
F
- Fact-checking
Verifying the factual accuracy of claims, either before publication within a news organisation or afterwards through public assessments of statements by politicians and others. Political fact-checking organisations grew rapidly from the 2000s and developed standard methods for sourcing and rating claims. (Graves, 2016; Kovach & Rosenstiel, 2014)
- Factor analysis
A family of methods that explain the correlations among many observed variables by a smaller number of unobserved (latent) factors. Exploratory factor analysis looks for structure; confirmatory factor analysis tests a structure specified in advance. Charles Spearman introduced the approach in 1904 in his work on general intelligence. (Spearman, 1904)
- FAIR data
Data that are Findable, Accessible, Interoperable and Reusable, according to guiding principles published in 2016. The principles emphasise rich metadata, persistent identifiers and clear licences so that both people and machines can find and reuse data. (Wilkinson et al., 2016)
- Fake review
An online review written to mislead, for example by a business praising itself or attacking competitors. A study of Yelp found that about 16% of restaurant reviews were filtered as suspicious, and that businesses with weak reputations or more competition were more likely to commit review fraud. (Luca & Zervas, 2016)
- Fallibilism
The view that any belief, however well supported, could turn out to be mistaken and should remain open to revision in light of new evidence. Fallibilism does not imply scepticism: it holds that we can have knowledge while admitting that our methods are imperfect. (Peirce, 1877; Popper, 1959)
- False confession
An admission of guilt for a crime the person did not commit. Research distinguishes voluntary, compliant (given to escape pressure) and internalised (the person comes to believe they did it) false confessions, and links them to lengthy, accusatory interrogation, vulnerable suspects and presentation of false evidence. (Kassin & Gudjonsson, 2004)
- False discovery rate also: FDR
The expected proportion of rejected null hypotheses that are actually true, that is, the share of "discoveries" that are false. The Benjamini–Hochberg procedure controls it when many tests are performed, offering more power than methods that control the chance of any false positive. (Benjamini & Hochberg, 1995)
- Falsifiability
Karl Popper's criterion for distinguishing scientific from non-scientific claims: a theory is scientific only if it forbids some possible observations, so that it could in principle be shown false by evidence. On this view science advances by bold conjectures and attempts at refutation rather than by accumulating confirmations. (Popper, 1959)
- Feminist inquiry
Approaches to research that examine how gender and power shape what counts as knowledge, who is treated as a knower and which questions are asked. Standpoint theory, developed by Sandra Harding and others, argues that starting from the lives of marginalised groups can produce more objective knowledge. (Harding, 1986)
- Field experiment
A randomised experiment conducted in a natural setting, such as a workplace, market or website, where participants often do not know they are in a study. Field experiments trade some control for greater realism and therefore stronger external validity than laboratory experiments. (Shadish et al., 2002)
- File-drawer problem
The tendency for studies with null or non-significant results to remain unpublished, "in the file drawer", so that the published literature overstates the size and reliability of effects. Robert Rosenthal named the problem in 1979 and proposed estimating how many unpublished null studies would be needed to overturn a finding. (Rosenthal, 1979; Franco et al., 2014)
- Filter bubble
Eli Pariser's term for the isolation that can result when personalisation algorithms show people mainly information that matches their past behaviour and views. Empirical studies suggest that its effects are smaller than often feared, with self-selection of sources playing a large role. (Pariser, 2011)
- Finding
A result of analysis, such as "42% of customers rate delivery as poor". A finding becomes an insight when it is interpreted: when it explains why the result occurs and what it means for a decision. The distinction matters because reports full of findings can leave decision-makers no clearer about what to do. (Ackoff, 1989; Rowley, 2007)
- Fishbone diagram also: Ishikawa diagram
A cause-and-effect diagram, introduced by Kaoru Ishikawa, that arranges possible causes of a problem along branches of categories, such as people, methods, machines, materials, measurement and environment. It helps teams consider causes broadly before testing which ones matter.
- Five forces also: Porter's five forces
Michael Porter's framework for analysing the competitive structure of an industry through five forces: rivalry among existing competitors, the threat of new entrants, the threat of substitutes, and the bargaining power of buyers and of suppliers. The combined strength of the forces shapes the industry's profit potential. (Porter, 1979)
- Five whys
A root-cause technique from the Toyota Production System in which one asks "why?" repeatedly, typically about five times, to move from a symptom to its underlying cause. Its simplicity is its strength and its weakness: it can stop at a single chain of causes when problems usually have several. (Ohno, 1988)
- Fixation of belief
Peirce's 1877 account of four methods by which people settle opinions: tenacity (holding to what one already believes), authority (accepting what institutions declare), the a priori method (accepting what seems agreeable to reason) and the method of science, which alone lets beliefs be corrected by something outside the believer. (Peirce, 1877)
- Forensic accounting
The application of accounting, auditing and investigative skills to legal questions such as fraud, embezzlement, financial statement manipulation, damages and disputes. Forensic accountants trace funds, analyse records and present findings suitable for use in court. (Hogan et al., 2008; Association of Certified Fraud Examiners, 2024)
- Forensic science
The application of scientific methods to questions of law, through disciplines such as DNA analysis, toxicology, fingerprint, firearm and digital examination. A 2009 US National Research Council report found that, apart from nuclear DNA analysis, many forensic disciplines lacked rigorous validation of their accuracy. (National Research Council, 2009)
- Foundationalism
The view that justified beliefs rest on a foundation of basic beliefs that do not need justification from other beliefs, such as beliefs about immediate experience. Coherentism, its main rival, holds that beliefs are justified by how well they fit together in a system. (BonJour, 1985)
- Framing effect
The finding that logically equivalent descriptions of a choice lead to different decisions, for example describing a treatment as saving 200 of 600 lives rather than letting 400 die. Framing shows that answers depend on how a question is asked, which matters for surveys, messaging and decisions. (Tversky & Kahneman, 1981)
- Fraud diamond
An extension of the fraud triangle by Wolfe and Hermanson that adds capability: the personal traits and position needed to recognise and exploit an opportunity to commit fraud. (Wolfe & Hermanson, 2004)
- Fraud triangle
A model, derived from Donald Cressey's 1953 study of embezzlers, holding that occupational fraud requires three conditions: pressure (a financial or other need), opportunity (weak controls) and rationalisation (a justification). It underlies auditing standards, although critics argue it neglects organisational and collective causes of fraud. (Cressey, 1953; Dorminey et al., 2012)
- Frequentist inference
An approach to statistics in which probability means long-run relative frequency, parameters are fixed but unknown, and methods are judged by how they would perform over repeated samples. Confidence intervals, p-values and hypothesis tests are frequentist tools. (Neyman, 1937)
G
- Generalised linear model also: GLM
An extension of linear regression, introduced by Nelder and Wedderburn in 1972, in which the outcome can follow any distribution from the exponential family (normal, binomial, Poisson and others) and is related to the predictors through a link function. Logistic and Poisson regression are special cases. (Nelder & Wedderburn, 1972)
- Gettier problem
The challenge, from Edmund Gettier's three-page 1963 paper, that a justified true belief can fail to be knowledge when it is true only by luck, for example a belief inferred from a false premise that happens to be correct. Attempts to add a fourth condition have generated a large literature. (Gettier, 1963)
- Grounded theory
A qualitative methodology, introduced by Barney Glaser and Anselm Strauss in 1967, for building theory from data rather than testing existing theory. It uses constant comparison between cases, open and focused coding, memo writing and theoretical sampling, continuing until new data no longer add to the emerging categories. (Glaser & Strauss, 1967)
- Groupthink
Irving Janis's term for a mode of thinking in cohesive groups where the desire for agreement overrides realistic appraisal of alternatives. Symptoms include an illusion of invulnerability, self-censorship, pressure on dissenters and stereotyped views of outsiders. Devil's advocacy, red teams and independent reviews are common remedies. (Janis, 1972)
H
- Hallucination
Output from an AI language model that is fluent and confident but not supported by its sources or by fact, such as an invented citation. Because hallucinations are hard to distinguish from accurate output, AI-assisted research and intelligence work requires verification of every factual claim against primary sources. (Ji et al., 2023; Bender et al., 2021)
- HARKing
Hypothesising After the Results are Known: presenting a hypothesis formed after seeing the data as if it had been stated in advance. It makes exploratory findings look confirmatory and inflates false-positive rates. Pre-registration makes it detectable. (Kerr, 1998)
- Hermeneutics also: hermeneutic inquiry
The theory and practice of interpretation, originally of texts and later of human action and experience. Hans-Georg Gadamer argued that understanding always starts from prejudgements and develops through a "fusion of horizons" between interpreter and text, in a cycle between the parts and the whole. (Gadamer, 1960/2004)
- Heuristic inquiry
A qualitative research approach developed by Clark Moustakas in which the researcher investigates an experience they have lived, moving through phases of initial engagement, immersion, incubation, illumination, explication and creative synthesis. (Moustakas, 1990)
- Heuristics and biases
A research programme begun by Amos Tversky and Daniel Kahneman showing that people judge probability and value with simple rules of thumb, such as representativeness, availability and anchoring, which are often useful but produce systematic errors. It is a major source of insight into consumer and organisational decisions. (Tversky & Kahneman, 1974; Kahneman, 2011)
- Hypothesis
A testable statement about the relationship between variables, derived from theory or prior observation, that research is designed to support or refute. A good hypothesis specifies the expected direction of an effect and could be shown false by the evidence. (Popper, 1959)
I
- Implicit Association Test also: IAT
A computer-based task that measures the strength of automatic associations between concepts by comparing how quickly people sort words or images when categories are paired in different ways. It is widely used to study implicit attitudes, although its reliability for individuals and its ability to predict behaviour are debated. (Greenwald et al., 1998)
- Imputation
Filling in missing values with plausible estimates so that complete-data methods can be used. Methods range from mean and hot-deck imputation to model-based and multiple imputation. Multiple imputation creates several completed datasets so that the extra uncertainty from missing data is reflected in standard errors. (Little & Rubin, 2019)
- Incubation
A period in which a problem is set aside after initial work, during which a solution may emerge. A meta-analysis found a reliable incubation effect, larger for creative problems with many possible solutions and when the break was filled with a light task. Graham Wallas made incubation the second of his four stages of creative thought. (Sio & Ormerod, 2009; Wallas, 1926)
- Indicators and warnings also: I&W
A method of monitoring for observable signs that a specified development, such as an attack, a market shift or a supplier failure, is becoming more likely. Analysts define in advance which indicators would signal change and track them systematically, which reduces the risk of surprise. (Heuer, 1999; Ansoff, 1975)
- Induction also: inductive reasoning
Reasoning from particular observations to general conclusions or predictions, such as inferring that all swans are white from many white swans. Inductive conclusions can be strong but never certain, and David Hume's problem of induction asks what justifies them at all. Popper's falsificationism was one response. (Popper, 1959; Walton, 2008)
- Information
Data organised and given meaning so that they are useful for a purpose. In Claude Shannon's technical sense, information is a reduction in uncertainty, measured in bits; in the semantic sense used in information science, it is meaningful, well-formed and, on some accounts, truthful data. (Shannon, 1948; Floridi, 2005)
- Information asymmetry
A situation in which one party to a transaction knows more than the other. George Akerlof's "market for lemons" showed that when buyers cannot tell good used cars from bad ones, good cars may be driven from the market. Independent research, ratings and warranties reduce asymmetry. (Akerlof, 1970)
- Information good
Anything that can be digitised, such as data, reports, software or music. Information goods are costly to produce but cheap to copy, and using them does not use them up, which leads to pricing by versioning and to the economic view of knowledge as partly a public good. (Shapiro & Varian, 1999; Arrow, 1962)
- Information literacy
The set of abilities needed to find, evaluate, use and create information effectively and ethically, including understanding that authority is constructed and contextual and that research is a process of inquiry. The Association of College and Research Libraries' 2015 framework is a standard reference. (Association of College and Research Libraries, 2015)
- Information overload
The state in which the amount of information exceeds a person's or organisation's capacity to process it, so that decision quality falls rather than rises with more information. Causes include volume, poor quality and poor organisation of information; remedies include filtering, prioritisation and better presentation. (Eppler & Mengis, 2004)
- Information-gap theory
Loewenstein's 1994 account of curiosity as a feeling of deprivation caused by a perceived gap in knowledge. It predicts that curiosity depends on knowing enough to notice the gap, and that it can be triggered deliberately by posing questions, presenting puzzles or revealing partial information. (Loewenstein, 1994)
- Inquiry
The process of moving from doubt or an unsettled situation to a settled, warranted belief through questioning, investigation and reasoning. Charles Sanders Peirce described inquiry as the struggle to escape the irritation of doubt; John Dewey defined it as the controlled transformation of an indeterminate situation into a unified one. (Peirce, 1877; Dewey, 1938)
- Inquiry-based learning
Teaching in which learners investigate questions, problems or scenarios rather than receiving information directly. Evidence suggests it works best when guided: minimally guided discovery tends to overload novices, while structured inquiry with scaffolding and feedback can be effective. (Kirschner et al., 2006; Hmelo-Silver, 2004)
- Insight
In psychology, a sudden restructuring of a problem that makes its solution apparent. In business, a new understanding of customers, markets or operations that explains behaviour and suggests what to do. Both senses share the idea of seeing something that was not evident in the data or the problem as first presented. (Kounios & Beeman, 2014; Duncker, 1945)
- Instrumental variable also: IV
A variable that affects the treatment, is unrelated to unmeasured confounders, and affects the outcome only through the treatment. Under these assumptions it can be used to estimate a causal effect from observational data, typically the effect for "compliers" whose treatment is changed by the instrument. (Angrist et al., 1996)
- Intellectual capital
The intangible knowledge assets of an organisation, often divided into human capital (people's skills and knowledge), structural capital (processes, systems, data and intellectual property) and relational capital (relationships with customers and partners). Resource-based theory treats such valuable, rare and hard-to-imitate resources as sources of competitive advantage. (Barney, 1991)
- Intellectual humility
Recognising that one's beliefs might be wrong and being attentive to the limits of one's evidence and knowledge. People higher in intellectual humility are more open to opposing views and better at judging the strength of arguments, according to research by Leary and colleagues. (Leary et al., 2017)
- Intelligence
In the organisational sense, information that has been collected, evaluated and analysed to give a decision-maker an advantage, together with the organisation and activity that produce it. Sherman Kent described intelligence as knowledge, organisation and activity. It differs from raw information in being evaluated and tailored to a decision. (Kent, 1949)
- Intelligence cycle
The conventional model of the intelligence process as a cycle of direction (setting requirements), collection, processing, analysis and production, and dissemination, followed by feedback. Critics argue that real intelligence work is less sequential, with analysts and policymakers interacting throughout. (Hulnick, 2006; Kent, 1949)
- Intelligence failure
A failure to anticipate or correctly assess a significant development, such as the attack on Pearl Harbor. Studies emphasise that failures usually stem less from missing information than from ambiguous signals, mistaken assumptions, organisational barriers and decision-makers' reluctance to act on warning, which is why some failures are unavoidable. (Betts, 1978)
- Intelligence requirement also: key intelligence question
A statement of the information a decision-maker needs, expressed as a question that collection and analysis can address. Key intelligence questions or topics prioritise effort, link the intelligence function to decisions and give a basis for measuring its value. (Hulnick, 2006)
- Internal validity
The extent to which a study supports the conclusion that a relationship between variables is causal, rather than the product of confounding, selection, history, maturation or other threats. Random assignment is the strongest protection; quasi-experimental designs address the threats with design features and analysis. (Shadish et al., 2002)
- Interpretivism
A philosophical position holding that social reality is constructed through the meanings people give to their actions, so that research must understand those meanings from the participants' point of view rather than only measure behaviour. It underpins much qualitative research, including ethnography and phenomenology. (Geertz, 1973; Lincoln & Guba, 1985)
- Inter-rater reliability
The degree to which different coders or raters give consistent judgements of the same material. Simple percentage agreement overstates it because some agreement occurs by chance; chance-corrected statistics such as Cohen's kappa or Krippendorff's alpha are preferred. (Cohen, 1960; Krippendorff, 2019)
- Investigative interviewing
Interviewing witnesses, victims and suspects with the aim of obtaining accurate and reliable information rather than a confession. It replaced accusatory interrogation in many countries after research linked coercive methods to false confessions. (Milne & Bull, 1999; Kassin & Gudjonsson, 2004)
- Item nonresponse
Missing answers to particular questions from people who otherwise took part in a survey, for example skipping an income question. It differs from unit nonresponse, where the whole interview is missing. It is usually handled by imputation or by analysis methods that make explicit assumptions about why the data are missing. (Little & Rubin, 2019; Groves et al., 2009)
J
- Jobs to be done also: JTBD
A framework that understands demand by asking what progress a customer is trying to make in a particular circumstance, the "job" for which they "hire" a product. It shifts attention from customer attributes and product features to the functional, social and emotional dimensions of the customer's situation. (Christensen et al., 2016)
- Justified true belief also: JTB
The traditional analysis, traced to Plato's Theaetetus, that a person knows a proposition if it is true, they believe it, and they are justified in believing it. Edmund Gettier's 1963 paper gave cases where all three conditions hold by luck, and most philosophers now accept that something more is needed. (Gettier, 1963)
K
- Key assumptions check
A structured technique in which analysts list the assumptions underlying a judgement, ask how confident they are in each and what would make it untrue, and identify which ones the conclusion depends on. It brings hidden premises into the open before they cause a failure. (Heuer, 1999)
- Key performance indicator also: KPI
A measure chosen to track progress towards an important objective, such as customer retention or cost per acquisition. Good KPIs are few, clearly defined, linked to strategy and balanced, as in the balanced scorecard's financial, customer, internal process and learning perspectives. Targets set on a measure can distort behaviour. (Kaplan & Norton, 1992)
- Knightian uncertainty also: radical uncertainty
Uncertainty that cannot be measured by known probabilities, as distinguished by Frank Knight in 1921 from risk, where the probabilities of outcomes can be estimated. The phrase "unknown unknowns" refers to the extreme case where the possible outcomes themselves are not known. (Knight, 1921)
- Know-how also: know-that
Practical knowledge of how to do something, distinguished by Gilbert Ryle in 1949 from know-that, knowledge of facts. Ryle argued that intelligent action is not just the application of known facts, a claim still debated in epistemology. (Ryle, 1949)
- Knowledge
In the classical philosophical analysis, justified true belief; the Gettier problem shows that this definition is incomplete. In organisations, knowledge is understanding that lets people act effectively, combining information with experience, context and judgement, and it includes both explicit and tacit forms. (Gettier, 1963; Nonaka, 1994)
- Knowledge economy
An economy in which production and services depend mainly on knowledge-intensive activities, intellectual capabilities and rapid technical change rather than physical inputs. Fritz Machlup first measured the share of the US economy devoted to producing and distributing knowledge in 1962. (Powell & Snellman, 2004; Machlup, 1962)
- Knowledge graph
A representation of knowledge as a graph of entities (such as people, places and concepts) connected by typed relationships, often with a schema or ontology defining the types. Knowledge graphs support search, data integration and reasoning, and are used by search engines and in enterprise data management. (Hogan et al., 2021)
- Knowledge management also: KM
The practices and systems by which organisations create, capture, store, share and apply knowledge, ranging from document repositories and expert directories to communities of practice and lessons-learned processes. Research suggests that technology alone rarely succeeds without attention to culture and incentives. (Alavi & Leidner, 2001)
- Knowledge transfer
The process by which knowledge moves from one person, team or organisation to another and is put to use. Transfer is harder for tacit, complex and context-specific knowledge, and depends on the receiver's absorptive capacity. (Cohen & Levinthal, 1990; Nonaka, 1994)
L
- Lasso
Least absolute shrinkage and selection operator: a regression method that adds a penalty proportional to the sum of the absolute values of the coefficients. The penalty shrinks coefficients towards zero and sets some exactly to zero, so it performs variable selection and reduces overfitting when there are many predictors. (Tibshirani, 1996; Hastie et al., 2009)
- Latent class analysis
A statistical model that assumes the population consists of a number of unobserved classes and estimates, from categorical survey answers, the probability of each answer in each class and each respondent's probability of class membership. It is a model-based alternative to cluster analysis for segmentation. (Hastie et al., 2009)
- Law of large numbers
The theorem that the average of independent observations from the same distribution converges to the expected value as the number of observations grows. It explains why large random samples give stable estimates, and it does not imply that short runs "even out", the gambler's fallacy. (Jaynes, 2003)
- Lean startup
An approach to building new businesses that treats a business plan as a set of hypotheses to be tested quickly with customers, using a build–measure–learn loop and a minimum viable product. Its aim is validated learning: evidence about what customers actually want before large investment. (Ries, 2011)
- Levels of measurement also: scales of measurement
Stanley Stevens's classification of variables as nominal (categories), ordinal (ordered categories), interval (equal differences, arbitrary zero) or ratio (equal differences, true zero). The level determines which statistics are meaningful, although applying the rules strictly, for example to rating scales, remains debated. (Stevens, 1946)
- Leverage-saliency theory
A theory of survey participation proposing that each feature of a survey request (topic, sponsor, incentive, burden) has a leverage, the weight a given person attaches to it, and a saliency, how prominent it is made in the request. A person's decision to take part depends on the combination, which is why the same design can attract some groups and deter others and why topic interest can produce nonresponse bias. (Groves et al., 2000; Groves et al., 2004)
- Lie detection also: deception detection
Attempts to tell truthful from deceptive statements by behaviour, speech or physiology. Research shows that people, including professionals, detect lies only slightly better than chance from nonverbal cues, and that methods focusing on the content and verifiability of statements are more promising. (Vrij, 2008)
- Likert scale
A multi-item measure introduced by Rensis Likert in 1932 in which respondents indicate their degree of agreement with a set of statements, and the item scores are summed or averaged into a scale score. In everyday use the term is often applied to a single agree–disagree item, which is more precisely a Likert-type item. Whether such data can be treated as interval rather than ordinal is a long-running debate. (Likert, 1932)
- Link analysis also: network mapping
Mapping the relationships between people, organisations, accounts, places and events as a network, to reveal structure, key actors and hidden connections. Social network analysis supplies measures such as centrality, though criminal network data are often incomplete and their boundaries unclear. (Sparrow, 1991)
- Logical fallacy
A pattern of reasoning that seems persuasive but fails to support its conclusion, either because of its form (formal fallacies such as affirming the consequent) or because of its content and context (informal fallacies such as ad hominem, straw man, false dilemma or appeal to authority). Recognising fallacies is a core skill of critical thinking. (Walton, 2008)
- Logistic regression
A regression model for a binary outcome in which the log-odds of the outcome is a linear function of the predictors. Exponentiated coefficients are odds ratios. It is a generalised linear model with a binomial distribution and a logit link. (Nelder & Wedderburn, 1972)
M
- Machine-generated knowledge
Claims, summaries or predictions produced by algorithms and AI models rather than directly by people. Language models can produce fluent text that is not grounded in fact, which raises questions about whether such systems know anything and requires human verification, transparency about sources and accountability. (Bender et al., 2021; Ji et al., 2023)
- Margin of error
The half-width of a confidence interval for a survey estimate, usually at 95% confidence. For a proportion from a simple random sample it is about 1.96 times the standard error. It reflects sampling error only: it says nothing about coverage, nonresponse or measurement error, and for nonprobability samples its meaning depends on model assumptions. (Lohr, 2021; Neyman, 1934)
- Market intelligence
Information and analysis about a market's size, growth, segments, customers, competitors, prices and trends, gathered from primary research, secondary sources and internal data to inform market entry, product and strategy decisions. (Chen et al., 2012)
- Measurement error
The difference between a recorded answer and the true value of what was meant to be measured, arising from question wording, respondents' memory and motives, interviewers, mode or data processing. Random measurement error adds noise and weakens correlations; systematic error biases estimates. (Groves & Lyberg, 2010; Tourangeau et al., 2000)
- Measurement invariance
The property that a measure assesses the same construct in the same way across groups, modes or time points. It is usually tested in steps (configural, metric, scalar) with multi-group confirmatory factor analysis. Without at least scalar invariance, differences in group means may reflect differences in how the questions work rather than true differences. (King et al., 2004)
- Member checking also: respondent validation
Returning data, interpretations or findings to participants to ask whether they recognise them as accurate accounts of their experience. Lincoln and Guba described it as the most crucial technique for establishing credibility, though participants' views may legitimately differ from the researcher's analysis. (Lincoln & Guba, 1985)
- Meta-analysis
A statistical method for combining the results of multiple studies of the same question into a pooled estimate of effect size, weighting each study by its precision. Fixed-effect models assume one true effect; random-effects models allow the true effect to vary between studies. Heterogeneity statistics and checks for publication bias are standard parts of the method. (Borenstein et al., 2009)
- Metacognition
Thinking about one's own thinking: knowledge of one's cognitive processes and the monitoring and control of them, such as noticing that one has not understood a passage. John Flavell introduced the concept in 1979. Good metacognition helps inquirers judge when they know enough and when to ask more. (Flavell, 1979)
- Metadata
Data that describe other data, such as a file's creation date, author, device or GPS location, or an email's routing headers. Metadata can establish when, where and by whom digital content was created, and it can also be altered, so it must be examined critically. (Casey, 2011)
- Methodical doubt
René Descartes's method, set out in his 1641 Meditations, of withholding assent from any belief that could possibly be doubted in order to find a foundation that is certain. It led to his conclusion "I think, therefore I am" and became a model for systematic scepticism in inquiry. (Descartes, 1641/1996)
- Missing at random also: MAR
In Rubin's classification of missing data, the condition that the probability a value is missing depends only on observed data, not on the missing value itself. Data are missing completely at random if missingness is unrelated to any data, and missing not at random if it depends on the unobserved value. Most imputation and weighting methods assume missing at random. (Little & Rubin, 2019)
- Mixed methods research
Research that deliberately combines qualitative and quantitative data and analysis within one study or programme, and integrates the two. Core designs are convergent (both collected together and compared), explanatory sequential (qualitative work explains quantitative results) and exploratory sequential (qualitative work informs a later quantitative phase). (Creswell & Plano Clark, 2018)
- Mixed-mode survey
A survey that collects data through more than one mode, such as web and telephone, either by offering a choice, by following up nonrespondents in a second mode, or by using different modes for different parts of the interview. Mixing modes can improve coverage and response but can introduce mode effects that make answers from different modes hard to compare. (de Leeuw, 2005; Dillman et al., 2014)
- Mode 2 knowledge production
A form of knowledge production described by Michael Gibbons and colleagues in 1994 that is organised around problems in a context of application, is transdisciplinary, involves diverse organisations and is subject to wider social accountability. It contrasts with Mode 1, discipline-based research within universities. (Gibbons et al., 1994)
- Mode effect
A difference in survey answers caused by the mode of data collection (face-to-face, telephone, mail, web, app) rather than by differences between respondents. Interviewer-administered modes tend to produce more socially desirable answers on sensitive topics; visual and aural presentation can change how response scales are used. (de Leeuw, 2005; Tourangeau & Yan, 2007)
- Model selection criteria also: AIC, BIC
Measures for choosing among statistical models that balance fit against complexity. The Akaike information criterion (AIC) estimates out-of-sample prediction error; the Bayesian information criterion (BIC) penalises extra parameters more heavily as sample size grows. Lower values indicate a preferred model. (Akaike, 1974)
- Monte Carlo simulation
Using repeated random sampling to approximate quantities that are hard to calculate exactly, such as the distribution of outcomes of a risky project or the behaviour of a statistical method. Metropolis and Ulam named and described the method in 1949, drawing on work at Los Alamos. (Metropolis & Ulam, 1949)
- MRP also: multilevel regression and post-stratification
Multilevel regression and post-stratification: a method that fits a multilevel model predicting a survey outcome from demographic and geographic variables, then weights the predictions for each population cell by its known size from census data. It produces estimates for small groups and areas and can correct for unrepresentative samples. (Park et al., 2004)
- Multilevel model also: hierarchical model
A regression model for data with a nested structure, such as employees within firms or repeated measures within people, that includes random effects at each level. It estimates variation within and between groups and "partially pools" estimates for small groups towards the overall mean. (Gelman & Hill, 2007)
- Multiple comparisons
The problem that when many hypothesis tests are performed, the chance that at least one gives a false positive grows well above the nominal significance level. Corrections such as Bonferroni (controlling the family-wise error rate) and Benjamini–Hochberg (controlling the false discovery rate) adjust for it. (Benjamini & Hochberg, 1995)
- Mystery shopping
A research method in which trained observers pose as ordinary customers to assess service quality, compliance or pricing. Because staff do not know they are being observed, it records actual behaviour, but it raises ethical questions about deception and consent. (Wilson, 2001)
N
- Natural experiment
A situation in which an external event, policy or rule assigns people to different conditions in a way that is as good as random, allowing causal effects to be studied without the researcher controlling assignment. Lotteries, arbitrary boundaries and sudden policy changes are common sources. (Angrist & Pischke, 2009; Card & Krueger, 1994)
- Net Promoter Score also: NPS
A customer loyalty metric based on one question asking how likely a customer is to recommend a company on a 0–10 scale. The score is the percentage of promoters (9–10) minus the percentage of detractors (0–6). It was popularised as a predictor of growth, but later research found it no better than other satisfaction and loyalty measures at predicting company revenue growth. (Reichheld, 2003; Keiningham et al., 2007)
- Netnography
Ethnographic research adapted to online communities and social media, introduced by Robert Kozinets for marketing research. It studies the shared meanings and practices of online groups through observation, participation and analysis of their communications, with attention to consent and privacy. (Kozinets, 2002)
- Neuromarketing
The use of neuroscience and physiological methods, such as brain imaging, EEG, eye tracking and skin conductance, to study consumer responses to products, brands and advertising. Reviews find that it can reveal processes that people cannot report, but that many commercial claims outrun the evidence. (Plassmann et al., 2012)
- Nonprobability sample
A sample in which units are selected by a process that does not give every member of the population a known, non-zero chance of selection, such as opt-in online panels, quota samples or convenience samples. Estimates from such samples rely on modelling assumptions, typically that weighting or matching variables account for how participants differ from the population. (Baker et al., 2013)
- Nonresponse bias
The difference between a survey estimate and the value it would have had with full response, caused by respondents differing from nonrespondents on the variable being measured. It depends on the correlation between response propensity and the survey variable, not on the response rate alone, which is why low-response surveys can be accurate and high-response surveys can be biased. (Groves, 2006; Groves & Peytcheva, 2008)
- Normal distribution also: Gaussian distribution
The symmetric, bell-shaped continuous distribution defined by its mean and standard deviation, with about 68% of values within one standard deviation of the mean and about 95% within two. Its importance comes largely from the central limit theorem. (Lohr, 2021)
- Norms of science also: CUDOS
Robert Merton's 1942 description of the ethos of science: communalism (findings are shared), universalism (claims are judged by impersonal criteria), disinterestedness (science serves knowledge, not personal gain) and organised scepticism (claims are scrutinised before acceptance). (Merton, 1973)
- Nudge
Any aspect of choice architecture that changes behaviour in a predictable way without forbidding options or significantly changing economic incentives, such as making a desirable option the default. Thaler and Sunstein popularised the term in 2008. (Thaler & Sunstein, 2008)
O
- Occam's razor also: principle of parsimony
The principle of parsimony: among explanations that account for the evidence equally well, prefer the one that makes fewer assumptions. It is a guide for choosing between hypotheses, not a guarantee that the simpler one is true. Model selection criteria such as AIC formalise a similar trade-off. (Lipton, 2004)
- Odds ratio
The ratio of the odds of an outcome in one group to the odds in another; a value of 1 means no association. It is the natural effect measure in logistic regression and case–control studies. When the outcome is common, an odds ratio exaggerates the corresponding risk ratio. (Nelder & Wedderburn, 1972)
- Open science
Practices that make research methods, data, materials, code and publications openly available so that others can check, reuse and build on them. It includes open access publishing, open data following the FAIR principles (findable, accessible, interoperable, reusable), pre-registration and shared analysis code. (Munafò et al., 2017; Wilkinson et al., 2016)
- Open-source intelligence also: OSINT
Intelligence derived from publicly available information, such as media, public records, company filings, academic work, satellite imagery and social media. The growth of the internet has made it the largest source for many questions; its main challenges are volume, verification and deception. (Glassman & Kang, 2012)
- Operationalisation
The process of turning an abstract concept into specific, observable measures or procedures, for example defining "customer loyalty" as repeat purchase within twelve months. Good operationalisation is explicit, so others can replicate the study, and valid, so the measure captures the concept. (Shadish et al., 2002)
- Organisational intelligence
An organisation's capacity to gather, interpret and use information about its environment to make decisions. Harold Wilensky's 1967 study showed how hierarchy, specialisation and secrecy distort the flow of intelligence in government and industry. (Wilensky, 1967; Huber, 1991)
- Organisational memory
Stored information from an organisation's history that can be brought to bear on present decisions, held in individuals, culture, procedures, structures and records. Knowledge is lost when it lives only in people who leave, which is why documentation and succession matter. (Walsh & Ungson, 1991)
- Overfitting
Fitting a model so closely to the particular sample that it captures noise as well as signal, so that it performs well on the training data but poorly on new data. It is the high-variance side of the bias–variance trade-off and is controlled by cross-validation, regularisation and simpler models. (Hastie et al., 2009)
- Overgeneralisation
Drawing broad conclusions from too few or unrepresentative cases. Experiments show that people generalise from a single vivid case even when told it is atypical, which is why anecdotes and small qualitative samples need to be checked against representative data before they guide decisions. (Hamill et al., 1980; Tversky & Kahneman, 1974)
P
- Paradata
Data about the process of collecting survey data, such as call records, timestamps, keystrokes, device type and interviewer observations. Paradata are used to monitor fieldwork, detect data quality problems, model response propensity and adjust for nonresponse. (Kreuter, 2013)
- Paradigm
In Thomas Kuhn's account, the shared set of theories, methods, exemplary problems and standards that defines normal science in a field. Accumulating anomalies can lead to a crisis and a scientific revolution in which one paradigm replaces another. In social research the word is also used for broad positions such as positivism, interpretivism and pragmatism. (Kuhn, 1962)
- PEACE model
A framework for investigative interviewing adopted in England and Wales in the early 1990s: Planning and preparation, Engage and explain, Account clarification and challenge, Closure, and Evaluation. It emphasises open questions, fairness and information gathering rather than confession seeking. (Milne & Bull, 1999)
- Peak–end rule
The finding that people judge a past experience largely by its most intense moment and its end, rather than by the sum of every moment. In one study, participants preferred to repeat a longer unpleasant experience that ended less painfully. It explains why service recoveries and endings matter disproportionately to customer satisfaction. (Kahneman et al., 1993)
- Peer review
The evaluation of research by independent experts before publication or funding. It is meant to catch errors and judge importance, but studies show it is slow, inconsistent between reviewers, poor at detecting fraud, and open to bias. Open review, registered reports and post-publication review are attempts to improve it. (Smith, 2006)
- Persona
A fictional but evidence-based profile of a typical user or customer segment, describing goals, behaviours, needs and frustrations. Alan Cooper popularised personas in software design. They are useful when built from research data and misleading when they encode assumptions or stereotypes. (Cooper, 1999)
- P-hacking
Analysing data in many ways (adding or dropping observations, trying different covariates, outcomes or subgroups) and reporting the analyses that reach statistical significance. Even without intent to deceive, such flexibility can make false-positive findings very likely. (Simmons et al., 2011; John et al., 2012)
- Phenomenology
A philosophical tradition and qualitative research approach that studies the structures of lived experience as it appears to the person, setting aside prior theories about it. Phenomenological research describes what an experience is like and what it means to those who have it. (Moustakas, 1990)
- Positivism
A philosophical position holding that genuine knowledge comes from observation and measurement, that the social world can be studied with the methods of the natural sciences, and that research should seek general, testable laws. Post-positivism keeps the commitment to empirical testing while accepting that all observation is fallible and theory-laden. (Popper, 1959; Kuhn, 1962)
- Post-truth
Describing circumstances in which objective facts are less influential in shaping public opinion than appeals to emotion and personal belief. It was Oxford Dictionaries' word of the year for 2016. Analyses link it to declining trust in institutions, partisan media and the speed of online information. (McIntyre, 2018)
- Potential outcomes also: Rubin causal model
A framework for causal inference, associated with Donald Rubin, in which each unit has a potential outcome under each possible treatment. The causal effect is the difference between them, and since only one is observed, the "fundamental problem of causal inference", effects are estimated by comparing groups under assumptions such as random assignment. (Rubin, 1974; Holland, 1986)
- Prebunking also: inoculation
Exposing people in advance to weakened examples of misinformation techniques, with explanations, so that they are better able to recognise and resist them later, on the model of a vaccine. Online games built on this idea have reduced the perceived reliability of fake news in experiments. (Roozenbeek & van der Linden, 2019; Lewandowsky et al., 2012)
- Prediction market
A market in which participants trade contracts whose payoff depends on a future event, so that prices can be read as aggregate probability estimates. Prediction markets have forecast elections and project outcomes well, though in forecasting tournaments well-aggregated team forecasts have matched or beaten them. (Wolfers & Zitzewitz, 2004; Atanasov et al., 2017)
- Pre-registration also: preregistration
Recording a study's hypotheses, design and analysis plan in a time-stamped public registry before data are collected or analysed. It separates confirmatory from exploratory analysis and makes p-hacking and HARKing visible. Clinical trial registration has been required by major medical journals since 2005. (Nosek et al., 2018; De Angelis et al., 2004)
- Principal component analysis also: PCA
A dimension-reduction method, first described by Karl Pearson in 1901, that transforms correlated variables into a smaller set of uncorrelated components that capture as much of the total variance as possible. Unlike factor analysis it does not model a latent cause of the variables. (Pearson, 1901)
- Printing revolution
The transformation in the production and spread of knowledge that followed Gutenberg's movable-type press in the mid-fifteenth century. Elizabeth Eisenstein argued that standardised, multiplied and preserved texts made cumulative scholarship, the Reformation and the Scientific Revolution possible. (Eisenstein, 1979; Burke, 2000)
- Prior probability also: prior
In Bayesian reasoning, the probability assigned to a hypothesis before the current evidence is considered, often based on base rates or earlier studies. Combined with the likelihood of the evidence, it gives the posterior probability. Neglecting the prior is the source of base rate neglect. (Bayes, 1763; Jaynes, 2003)
- Probability sample
A sample drawn so that every unit in the population has a known, non-zero probability of selection. These probabilities allow design-based inference: unbiased estimation with weights equal to the inverse of the selection probabilities, and measurable sampling error. Simple random, systematic, stratified and cluster sampling are all forms of probability sampling. (Neyman, 1934; Kish, 1965; Lohr, 2021)
- Problem-based learning also: PBL
An approach in which students learn by working in groups on complex, realistic problems, identifying what they need to know and researching it, with a tutor who facilitates rather than lectures. It originated in medical education at McMaster University in the late 1960s. (Hmelo-Silver, 2004)
- Productive failure
A learning design in which students first attempt problems beyond their current knowledge, usually failing to solve them, before receiving instruction. Manu Kapur's studies found that this sequence can produce deeper conceptual understanding than instruction first. (Kapur, 2008)
- Propensity score
The probability that a unit receives a treatment given its observed characteristics. Rosenbaum and Rubin showed that matching, weighting or stratifying on this single score can balance the observed covariates between treated and untreated groups. It does not correct for unmeasured confounders. (Rosenbaum & Rubin, 1983)
- Provenance
A record of the origin and history of a piece of data or knowledge: who created it, from what sources, by what processes and with what changes. Provenance lets users judge reliability and reproduce results. The W3C PROV standard models it in terms of entities, activities and agents. (Moreau & Missier, 2013; Wilkinson et al., 2016)
- Psychological safety
A shared belief within a team that it is safe to take interpersonal risks, such as asking questions, admitting mistakes or raising concerns. Amy Edmondson found that psychological safety supports team learning behaviour, which in turn improves performance. (Edmondson, 1999)
- Publication bias
Systematic distortion of the published record that occurs when the chance of a study being published depends on the direction or significance of its results. It inflates average effect sizes in literature reviews and meta-analyses. Funnel plots and comparisons with registered studies are used to detect it. (Franco et al., 2014; Rosenthal, 1979)
- P-value
The probability, computed assuming the null hypothesis and all model assumptions are true, of obtaining a result at least as extreme as the one observed. It is not the probability that the null hypothesis is true, nor a measure of effect size. The American Statistical Association's 2016 statement warns against basing conclusions on whether p crosses 0.05. (Wasserstein & Lazar, 2016; Greenland et al., 2016)
Q
- Qualitative comparative analysis also: QCA
A case-oriented method developed by Charles Ragin that uses Boolean algebra and set theory to identify which combinations of conditions are sufficient or necessary for an outcome across a small to medium number of cases. It is suited to causal complexity, where different combinations of conditions can lead to the same outcome. (Ragin, 1987)
- Quasi-experiment
A study that tests a causal hypothesis by comparing outcomes under different conditions but without random assignment to those conditions. Designs such as non-equivalent control groups, interrupted time series and regression discontinuity use comparison groups, repeated measures and design features to rule out alternative explanations. (Shadish et al., 2002)
- Question Formulation Technique also: QFT
A structured process, developed by the Right Question Institute, in which learners produce as many questions as they can about a prompt without stopping to discuss them, convert questions between open and closed forms, prioritise them and plan how to use them. (Rothstein & Santana, 2011)
- Questionable research practices also: QRPs
Practices that fall short of fabrication or falsification but inflate false-positive rates, such as selectively reporting outcomes, deciding to collect more data after looking at results, excluding data after seeing their effect, and presenting unexpected findings as predicted. A large survey of psychologists found such practices to be common. (John et al., 2012; Simmons et al., 2011)
- Quota sampling
A nonprobability method in which interviewers or online systems fill target numbers (quotas) of respondents in categories such as age, sex and region so that the sample matches the population on those variables. Selection within each quota cell is not random, so the sample can still differ from the population on unmeasured characteristics. Quota polls famously failed in the 1948 US presidential election. (Baker et al., 2013; Converse, 1987)
R
- Randomised controlled trial also: RCT
An experiment in which participants are randomly allocated to an intervention or a control condition and outcomes are compared. Randomisation balances known and unknown confounders in expectation, which is why RCTs are regarded as the strongest design for estimating causal effects when they are feasible and ethical. (Shadish et al., 2002; Rubin, 1974)
- Randomized response technique
A method for asking sensitive questions in which a random device, unseen by the interviewer, decides whether the respondent answers the sensitive question or an innocuous one. Because no single answer reveals the respondent's status, people may answer more honestly, while the known probabilities still allow the prevalence of the sensitive trait to be estimated. (Warner, 1965; Tourangeau & Yan, 2007)
- Realist evaluation
An approach to evaluation that asks what works, for whom, in what circumstances and why. It explains outcomes through context–mechanism–outcome configurations: programmes work by triggering mechanisms in people, and whether they do depends on context. (Pawson & Tilley, 1997)
- Red teaming also: devil's advocacy
Assigning a group to take the perspective of an adversary or to challenge a plan, so that weaknesses and alternative outcomes are exposed before they are exploited. Devil's advocacy, in which one person argues against the prevailing view, is a lighter form of the same idea. (Zenko, 2015; Heuer, 1999)
- Reflexivity
The researcher's ongoing, explicit examination of how their own background, assumptions, position and relationships shape the questions asked, the data gathered and the interpretations made. It is a standard of quality in qualitative and feminist inquiry. (Harding, 1986; Lincoln & Guba, 1985)
- Registered report
A journal article format in which the introduction, methods and analysis plan are peer reviewed before data are collected. If the protocol is accepted in principle, the journal commits to publishing the results whatever they show, which removes the incentive to produce positive findings. Registered reports report far fewer positive results than standard articles. (Chambers, 2013; Scheel et al., 2021)
- Regression
A family of methods that model how the expected value of an outcome changes with one or more predictors. In linear regression, coefficients estimate the change in the outcome per unit change in a predictor, holding the other predictors constant. Galton coined the term from his observation of regression to the mean in heredity. (Galton, 1886; Gelman & Hill, 2007)
- Regression discontinuity design also: RDD
A quasi-experimental design used when treatment is assigned by whether a score crosses a cut-off, such as a test mark for a scholarship. Comparing units just above and just below the threshold estimates the causal effect at the cut-off, because those units are otherwise similar. (Thistlethwaite & Campbell, 1960; Angrist & Pischke, 2009)
- Regression to the mean
The tendency for unusually high or low measurements to be followed by values closer to the average, because extreme values partly reflect chance. Francis Galton described it in heredity. Ignoring it leads people to credit interventions, or blame them, for changes that would have happened anyway. (Galton, 1886; Kahneman & Tversky, 1973)
- Reliability
The consistency of a measurement: the degree to which it gives the same result under the same conditions. Forms include test–retest reliability (stability over time), internal consistency (agreement among items, often measured by Cronbach's alpha) and inter-rater reliability (agreement among coders). A measure can be reliable without being valid. (Cronbach, 1951)
- Replication crisis
The finding, from large-scale replication projects in psychology, economics, medicine and other fields since about 2011, that many published results cannot be reproduced. In one project only about 36% of 100 psychology studies produced a significant result again. Causes include small samples, publication bias and flexible analysis; reforms include pre-registration and open data. (Open Science Collaboration, 2015; Ioannidis, 2005; Munafò et al., 2017)
- Research misconduct also: FFP
Fabrication (making up data), falsification (manipulating data, materials or processes) or plagiarism (using others' work without credit) in proposing, performing, reviewing or reporting research. A meta-analysis of surveys found that about 2% of scientists admitted to having fabricated or falsified data at least once. (Fanelli, 2009)
- Respondent-driven sampling also: RDS
A network-based method for sampling hidden or hard-to-reach populations, in which initial participants (seeds) recruit peers, who recruit further peers in successive waves. Recruitment is tracked and weighted by participants' network sizes to reduce the bias of chain referral, under assumptions that are often hard to verify in practice. (Heckathorn, 1997)
- Response rate
The proportion of eligible sample members who complete a survey. AAPOR's Standard Definitions set out several formulas that differ in how partial interviews and cases of unknown eligibility are treated. A low response rate raises the risk of nonresponse bias but does not by itself show that bias is present. (American Association for Public Opinion Research, 2023; Groves & Peytcheva, 2008)
- Responsive design
A survey design that sets out in advance how fieldwork will change in phases, based on paradata and early results, to control costs and errors; for example switching mode or raising incentives for groups with low response. Adaptive design is a related approach that tailors protocols to sample members from the start. (Groves & Heeringa, 2006)
- Restructuring
The Gestalt psychologists' term for the change in how a problem is represented that produces insight, for example seeing an object's unusual use. Karl Duncker's studies of functional fixedness, such as the candle problem, showed how prior use of an object blocks restructuring. (Duncker, 1945; Kounios & Beeman, 2014)
- Root cause analysis also: RCA
A structured investigation of an incident or problem that seeks the underlying causes, in processes, systems and organisation, rather than stopping at the immediate error. Methods include the five whys, fishbone diagrams, fault tree analysis and timeline reconstruction. (Reason, 2000; Ohno, 1988)
S
- Sampling distribution
The distribution of a statistic, such as a sample mean, across all possible samples of the same size from a population. Its standard deviation is the standard error. It is the basis of confidence intervals and hypothesis tests. (Lohr, 2021)
- Sampling frame
The list or procedure used to identify and select members of the target population, such as an address file, a telephone number range or a customer database. The quality of a sample can be no better than its frame: units missing from the frame cannot be selected, which leads to coverage error. (Groves et al., 2009; Lohr, 2021)
- Satisficing
Giving answers that are merely acceptable rather than optimal, because carefully completing each step of the response process is demanding. Weak satisficing includes choosing the first reasonable option or agreeing with statements; strong satisficing includes straightlining, choosing "don't know" or random answering. It is more likely when the task is difficult, motivation is low and ability is limited. (Krosnick, 1991)
- Saturation
The point in qualitative data collection at which new interviews or observations no longer produce new themes or insights relevant to the research question. In one widely cited study, most themes appeared within the first twelve interviews in a fairly homogeneous group, though the number needed depends on the aims, population and analysis. (Guest et al., 2006; Glaser & Strauss, 1967)
- Say–do gap also: attitude–behaviour gap
The difference between what people say they think, intend or will do and what they actually do. Meta-analyses show that intentions explain only part of the variation in behaviour, and LaPiere's 1934 study found that stated refusals to serve Chinese guests contrasted sharply with actual service. Insight work therefore combines stated and observed data. (LaPiere, 1934; Sheeran, 2002)
- Scenario planning
A method for exploring several plausible futures, built from the key uncertainties and driving forces affecting an organisation, so that strategies can be tested against each. Its aim is to widen thinking and prepare for change rather than to predict a single outcome. (Schoemaker, 1995)
- Scientific Revolution
The transformation of natural knowledge in Europe between roughly the mid-sixteenth and late seventeenth centuries, associated with Copernicus, Galileo, Kepler, Bacon, Descartes, Boyle and Newton, and with new institutions such as the Royal Society. Historians now stress its gradual, contested and social character. (Shapin, 1996)
- Scoping review
A review that maps the extent, range and nature of research on a broad topic, identifies key concepts and gaps, and often precedes a systematic review. It follows a transparent search process but does not usually appraise study quality or pool results. (Arksey & O'Malley, 2005)
- SECI model
Ikujiro Nonaka's model of organisational knowledge creation as a spiral of four conversions between tacit and explicit knowledge: socialisation (tacit to tacit), externalisation (tacit to explicit), combination (explicit to explicit) and internalisation (explicit to tacit). (Nonaka, 1994; Nonaka & Takeuchi, 1995)
- Segmentation also: market segmentation
Dividing a heterogeneous market into smaller groups of customers with similar needs, characteristics or behaviour, so that products and marketing can be targeted to each. Wendell Smith introduced the concept in 1956. Segments are commonly derived from survey data with cluster or latent class analysis. (Smith, 1956)
- Selection bias
Bias that arises when the units included in an analysis differ systematically from the population of interest because of how they were selected, or because selection depends on both exposure and outcome. Coverage error, nonresponse bias, survivorship bias and collider bias are forms of it. (Hernán et al., 2004)
- Semiotics
The study of signs and how they create meaning. In consumer research it analyses how products, packaging, advertising and brands communicate through codes, symbols and cultural conventions, and how consumers read them. (Mick, 1986)
- Sensemaking
Karl Weick's term for the ongoing process by which people and organisations give meaning to ambiguous or surprising events, by noticing cues, placing them in frameworks and acting on plausible interpretations. It emphasises that understanding is constructed in action and in conversation, not simply discovered in data. (Weick, 1995)
- Serendipity
Discovery by a combination of accident and sagacity: finding something valuable that was not sought, and recognising its significance. Horace Walpole coined the word in 1754; Robert Merton studied its role in science, emphasising the prepared mind needed to notice an anomaly. (Merton & Barber, 2004)
- Simpson's paradox
A reversal in which an association between two variables in every subgroup disappears or reverses when the subgroups are combined. The Berkeley graduate admissions data are a classic example. Which comparison is correct depends on the causal structure, not on the arithmetic. (Simpson, 1951; Bickel et al., 1975; Pearl, 2009)
- Small area estimation
Methods for producing reliable estimates for geographic areas or groups where the survey sample is too small for direct estimates. Models borrow strength from other areas and from auxiliary data such as census or administrative records. (Rao & Molina, 2015)
- Snowball sampling also: chain-referral sampling
A nonprobability method in which initial participants refer others they know who meet the study criteria, who in turn refer more. It is useful for reaching hidden or rare populations, but the sample reflects the social networks of the starting points. Respondent-driven sampling adds tracking and weighting to reduce this bias. (Heckathorn, 1997)
- Social epistemology
The study of the social dimensions of knowledge: how testimony, institutions, expertise, communication systems and collective deliberation help or hinder groups and individuals in forming true beliefs. Alvin Goldman framed it as evaluating social practices by their tendency to produce truth. (Goldman, 1999)
- Socratic questioning
A disciplined method of asking questions to examine ideas, uncover assumptions, test the logic and evidence behind beliefs and follow their implications. Derived from the dialogues of Plato's Socrates, it is used in teaching, coaching, therapy and analysis. (Paul & Elder, 2006)
- Source evaluation
Assessing information by the reliability of its source and the credibility of its content, often with a two-part grading such as the NATO Admiralty code (A–F for source reliability, 1–6 for information credibility). Research shows such ratings are often inconsistent and that analysts should consider a source's access and corroboration explicitly. (Irwin & Mandel, 2019)
- Spurious correlation
An association between two variables that does not reflect a causal link between them, because both are driven by a third factor, because of chance, because of shared trends over time, or because of how the variables were constructed. Karl Pearson used the term in 1897 for correlations between ratios with a common denominator. (Pearson, 1897; Yule, 1926; Messerli, 2012)
- Standard deviation
A measure of spread equal to the square root of the variance, the average squared distance of values from their mean. It is expressed in the units of the data. For roughly normal data, about two-thirds of values lie within one standard deviation of the mean. (Tukey, 1977)
- Standard error
The standard deviation of the sampling distribution of a statistic, measuring how much the statistic would vary from sample to sample. For a mean from a simple random sample it equals the standard deviation divided by the square root of the sample size. Complex designs require design-based variance estimation. (Lohr, 2021; Kish, 1965)
- Statistical power
The probability that a test will detect an effect of a given size if it exists, that is, one minus the probability of a Type II error. Power depends on the effect size, the sample size, the significance level and the variability of the data. Underpowered studies miss real effects and exaggerate the effects they do find. (Cohen, 1988)
- Statistical process control also: SPC
Monitoring a process over time with control charts that separate common-cause variation, inherent in a stable process, from special-cause variation that signals a change worth investigating. Walter Shewhart developed the method at Bell Laboratories in the 1920s. (Shewhart, 1931)
- Statistical significance
The label applied when a p-value falls below a chosen threshold, conventionally 0.05. A significant result is not necessarily important, and a non-significant result is not evidence of no effect. Many statisticians recommend reporting effect sizes and confidence intervals instead of dichotomising results. (Wasserstein & Lazar, 2016; Greenland et al., 2016)
- Storytelling with data
Presenting analytical results as a narrative with a clear audience, message and structure, supported by visuals designed to make the key point obvious. The aim is to help decision-makers understand and remember what the data mean, without distorting the evidence. (Knaflic, 2015; Tufte, 1983)
- Straightlining also: nondifferentiation
Giving the same answer to every item in a grid or battery of questions regardless of content. It is a common indicator of satisficing or careless responding, although identical answers can sometimes be genuine. Shorter grids, item-by-item presentation and varied item wording reduce it. (Krosnick, 1991; Meade & Craig, 2012)
- Stratified sampling
A probability design in which the population is divided into non-overlapping groups (strata) and a separate sample is drawn from each. Stratification guarantees representation of each stratum and, when strata differ on the survey variable, increases precision. Neyman (1934) showed how to allocate the sample across strata to minimise variance. (Neyman, 1934; Lohr, 2021)
- Structured analytic techniques also: SATs
Methods that make an analyst's reasoning explicit and open to review, such as analysis of competing hypotheses, key assumptions checks, red teaming, indicators and scenarios. They were promoted to counter the cognitive biases Heuer described, although evidence on how much they improve accuracy is mixed. (Heuer, 1999)
- Superforecasting
The practice of the top performers in the Good Judgment Project, a forecasting tournament run from 2011 to 2015, whose probability forecasts on geopolitical questions were markedly more accurate than those of other participants. Accuracy was linked to cognitive style, training in probabilistic reasoning, teamwork and frequent small updates. (Tetlock & Gardner, 2015; Mellers et al., 2015)
- Survey response process
The model of answering a survey question as four components: comprehension of the question, retrieval of relevant information from memory, judgement to combine that information into an answer, and response, mapping the answer onto the categories offered. Errors can enter at each stage; the model underpins cognitive interviewing and much questionnaire design advice. (Tourangeau et al., 2000)
- Survivorship bias
Drawing conclusions only from cases that passed a selection process, such as successful firms or returning aircraft, while overlooking those that did not. Abraham Wald's Second World War analysis of where to armour bombers, based on where surviving planes were not hit, is the classic example. (Mangel & Samaniego, 1984)
- Swiss cheese model
James Reason's model of accident causation in which an organisation's defences are slices of cheese with holes (active failures and latent conditions). An accident happens when holes in successive layers line up. It directs investigators towards system weaknesses rather than individual blame. (Reason, 2000; Reason, 1990)
- SWOT analysis
A planning framework that lists an organisation's internal strengths and weaknesses and its external opportunities and threats. It is easy to use and communicate but is often criticised for producing unprioritised lists; it is most useful when each item is backed by evidence and linked to decisions. (Porter, 1979)
- Synthetic control method
A method for estimating the effect of an intervention on a single unit, such as a region, by constructing a weighted combination of untreated units that closely matches the treated unit before the intervention. The divergence between the treated unit and its synthetic control afterwards estimates the effect. (Abadie et al., 2010)
- Systematic review
A review that answers a defined question by identifying, appraising and synthesising all relevant studies using explicit, reproducible methods: a protocol, comprehensive searches, predefined inclusion criteria and risk-of-bias assessment. The PRISMA 2020 statement sets out how such reviews should be reported. (Page et al., 2021)
- Systems thinking
An approach to inquiry that looks at how the parts of a system interact through stocks, flows, feedback loops and delays to produce behaviour over time, rather than examining parts in isolation. It helps explain why interventions in complex systems often have unintended effects. (Meadows, 2008)
T
- Tacit knowledge
Knowledge that people have but cannot fully put into words, such as how to ride a bicycle or recognise a face. Michael Polanyi summarised it as "we can know more than we can tell". Tacit knowledge is transferred mainly by shared practice and is hard to capture in documents. (Polanyi, 1966; Nonaka, 1994)
- Telescoping
A recall error in which respondents report events as having happened more recently (forward telescoping) or longer ago (backward telescoping) than they did, moving them into or out of the reference period of a question. Bounded interviewing, where a previous interview marks the start of the period, was developed to reduce it. (Neter & Waksberg, 1964; Tourangeau et al., 2000)
- Testimony
Knowledge gained from what others tell us, through conversation, books, experts, surveys and media. Most of what any person knows depends on it. Philosophers debate whether testimony is a basic source of justification or must be backed by evidence of the speaker's reliability. (Coady, 1992; Goldman, 1999)
- Thematic analysis
A method for identifying, analysing and reporting patterns of meaning (themes) in qualitative data. Braun and Clarke's widely used version has six phases: familiarisation, generating codes, constructing themes, reviewing themes, defining and naming themes, and writing up. (Braun & Clarke, 2006)
- Thought experiment
An imagined scenario used to test a theory, clarify a concept or reveal a hidden assumption by reasoning about what would happen, as in Galileo's falling bodies or Einstein's train. Thought experiments are a form of inquiry that relies on reasoning rather than new observation. (Sorensen, 1992)
- Total survey error also: TSE
A framework that treats survey quality as the sum of all errors affecting an estimate: coverage, sampling, nonresponse and adjustment errors on the representation side, and validity, measurement and processing errors on the measurement side. It encourages designers to balance errors against costs rather than focusing on sampling error alone. (Groves & Lyberg, 2010; Groves et al., 2009)
- Triangulation
Using more than one method, data source, investigator or theory to study the same phenomenon, so that conclusions do not depend on the weaknesses of a single approach. Norman Denzin distinguished data, investigator, theory and methodological triangulation. (Denzin, 1978)
- Triple helix
A model of innovation in which universities, industry and government interact, each taking on some roles of the others, for example universities founding companies. Etzkowitz and Leydesdorff proposed it as an alternative to models centred on firms or on national systems. (Etzkowitz & Leydesdorff, 2000)
- Trustworthiness
Lincoln and Guba's criteria for judging qualitative research, parallel to validity and reliability in quantitative work: credibility (confidence in the findings), transferability (applicability to other contexts), dependability (consistency of the process) and confirmability (grounding in the data rather than the researcher's bias). (Lincoln & Guba, 1985)
- t-test also: Student's t-test
A test comparing a mean with a value, or two means with each other, using the t distribution, which accounts for the extra uncertainty from estimating the standard deviation in small samples. William Sealy Gosset published it in 1908 under the pseudonym "Student". (Student, 1908)
- Tunnel vision
In criminal justice, the process by which investigators focus on a suspect early and then select and filter evidence in ways that build a case against that person while ignoring evidence pointing elsewhere. It is a leading contributor to wrongful convictions and is driven by confirmation bias, hindsight bias and institutional pressures. (Findley & Scott, 2006)
- Type I error
Rejecting a null hypothesis that is true, a false positive. Its probability is set by the significance level, often 0.05. A Type II error is failing to reject a false null hypothesis, a false negative; its probability is one minus the power. (Cohen, 1988)
U
- Unit nonresponse
The failure to obtain any survey data from a sampled unit, because the person or organisation could not be contacted, refused, or was unable to take part. It is distinguished from item nonresponse and is the main source of declining response rates in household surveys. (Groves & Couper, 1998; Brick, 2013)
V
- Value of information
In decision theory, the increase in expected value of a decision that new information would bring, compared with deciding without it. Ronald Howard formalised it in 1966. It shows that information is worth buying only if it could change the decision, and that the expected value of perfect information sets an upper limit on what to pay for research. (Howard, 1966)
- Voice of the customer also: VoC
A structured process for capturing customers' needs and preferences in their own words, through interviews, surveys, complaints and feedback, and translating them into product and service requirements. Griffin and Hauser's 1993 study showed that a modest number of interviews uncovers most customer needs. (Griffin & Hauser, 1993)
W
- Wallas's four stages
Graham Wallas's 1926 model of creative thought as preparation (working on the problem), incubation (setting it aside), illumination (the solution appears) and verification (testing it). The model remains a common framework for research on insight and creativity. (Wallas, 1926)
- Weak signal
An early, imprecise indication of a change that may become significant. Igor Ansoff argued in 1975 that firms should respond in graduated steps to weak signals of strategic surprise rather than wait for full information. Horizon scanning is the systematic search for such signals. (Ansoff, 1975)
- Weighted mean
An average in which each value counts in proportion to a weight, calculated as the sum of each value times its weight divided by the sum of the weights. In surveys the weights represent how many population members each respondent stands for, so the weighted mean estimates the population mean. (Horvitz & Thompson, 1952; Kish, 1992)
- Weighting also: survey weights
Assigning each respondent a weight so that weighted estimates represent the target population. Base weights correct for unequal selection probabilities; nonresponse and calibration adjustments (post-stratification, raking, generalised regression) make weighted totals match known population figures. Weighting can reduce bias but usually increases variance. (Kalton & Flores-Cervantes, 2003; Deville & Särndal, 1992; Horvitz & Thompson, 1952)
- Whistleblowing
The disclosure by members of an organisation of illegal, immoral or illegitimate practices under the control of their employers to people or bodies that may be able to act. Internal reporting channels, protection from retaliation and proper investigation of reports determine whether whistleblowing helps an organisation correct wrongdoing. (Near & Miceli, 1985)
- Wicked problem
Horst Rittel and Melvin Webber's 1973 term for social and policy problems that have no definitive formulation, no stopping rule, no true-or-false solutions and no way to test solutions without consequences, such as poverty or climate policy. Such problems call for iterative inquiry and stakeholder involvement rather than a single optimal solution. (Rittel & Webber, 1973)
- Wisdom of crowds
The finding that the aggregate of many independent judgements is often more accurate than most individual judgements, as in Francis Galton's 1907 report that the median guess of an ox's weight at a fair was within 1% of the true value. It depends on diversity and independence of opinions; when judgements are correlated or influenced by each other the advantage shrinks. (Galton, 1907; Surowiecki, 2004)
- Words of estimative probability also: WEP
Verbal expressions such as "likely" or "almost certain" used in intelligence estimates to convey probability. Sherman Kent showed in 1964 that readers interpret such words very differently and proposed linking each to a numerical range. Many intelligence agencies now publish standard probability yardsticks. (Kent, 1964; Mandel & Barnes, 2014)
References
Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic control methods for comparative case studies: Estimating the effect of California's tobacco control program. Journal of the American Statistical Association, 105(490), 493–505. https://doi.org/10.1198/jasa.2009.ap08746
Ackoff, R. L. (1989). From data to wisdom. Journal of Applied Systems Analysis, 16, 3–9.
Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/TAC.1974.1100705
Akerlof, G. A. (1970). The market for "lemons": Quality uncertainty and the market mechanism. The Quarterly Journal of Economics, 84(3), 488–500. https://doi.org/10.2307/1879431
Alavi, M., & Leidner, D. E. (2001). Review: Knowledge management and knowledge management systems: Conceptual foundations and research issues. MIS Quarterly, 25(1), 107–136. https://doi.org/10.2307/3250961
American Association for Public Opinion Research. (2023). Standard definitions: Final dispositions of case codes and outcome rates for surveys (10th ed.). https://aapor.org/wp-content/uploads/2023/05/Standards-Definitions-10th-edition.pdf
Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. Longman.
Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press. https://doi.org/10.1515/9781400829828
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434), 444–455. https://doi.org/10.1080/01621459.1996.10476902
Anscombe, F. J. (1973). Graphs in statistical analysis. The American Statistician, 27(1), 17–21. https://doi.org/10.1080/00031305.1973.10478966
Ansoff, H. I. (1975). Managing strategic surprise by response to weak signals. California Management Review, 18(2), 21–33. https://doi.org/10.2307/41164635
Argyris, C., & Schön, D. A. (1978). Organizational learning: A theory of action perspective. Addison-Wesley.
Arksey, H., & O'Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32. https://doi.org/10.1080/1364557032000119616
Arrow, K. J. (1962). Economic welfare and the allocation of resources for invention. In National Bureau of Economic Research (Ed.), The rate and direction of inventive activity: Economic and social factors (pp. 609–626). Princeton University Press.
Association of Certified Fraud Examiners. (2024). Occupational fraud 2024: A report to the nations. https://www.acfe.com/RTTN
Association of College and Research Libraries. (2015). Framework for information literacy for higher education. American Library Association. https://www.ala.org/acrl/standards/ilframework
Atanasov, P., Rescober, P., Stone, E., Swift, S. A., Servan-Schreiber, E., Tetlock, P., Ungar, L., & Mellers, B. (2017). Distilling the wisdom of crowds: Prediction markets vs. prediction polls. Management Science, 63(3), 691–706. https://doi.org/10.1287/mnsc.2015.2374
Baker, R., Brick, J. M., Bates, N. A., Battaglia, M., Couper, M. P., Dever, J. A., Gile, K. J., & Tourangeau, R. (2013). Summary report of the AAPOR task force on non-probability sampling. Journal of Survey Statistics and Methodology, 1(2), 90–143. https://doi.org/10.1093/jssam/smt008
Bar-Hillel, M. (1980). The base-rate fallacy in probability judgments. Acta Psychologica, 44(3), 211–233. https://doi.org/10.1016/0001-6918(80)90046-3
Barney, J. (1991). Firm resources and sustained competitive advantage. Journal of Management, 17(1), 99–120. https://doi.org/10.1177/014920639101700108
Bayes, T. (1763). An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 53, 370–418. https://doi.org/10.1098/rstl.1763.0053
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610–623). ACM. https://doi.org/10.1145/3442188.3445922
Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x
Berkson, J. (1946). Limitations of the application of fourfold table analysis to hospital data. Biometrics Bulletin, 2(3), 47–53. https://doi.org/10.2307/3002000
Betts, R. K. (1978). Analysis, war, and decision: Why intelligence failures are inevitable. World Politics, 31(1), 61–89. https://doi.org/10.2307/2009967
Bickel, P. J., Hammel, E. A., & O'Connell, J. W. (1975). Sex bias in graduate admissions: Data from Berkeley. Science, 187(4175), 398–404. https://doi.org/10.1126/science.187.4175.398
BonJour, L. (1985). The structure of empirical knowledge. Harvard University Press.
Bonney, R., Cooper, C. B., Dickinson, J., Kelling, S., Phillips, T., Rosenberg, K. V., & Shirk, J. (2009). Citizen science: A developing tool for expanding science knowledge and scientific literacy. BioScience, 59(11), 977–984. https://doi.org/10.1525/bio.2009.59.11.9
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Wiley. https://doi.org/10.1002/9780470743386
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Brick, J. M. (2013). Unit nonresponse and weighting adjustments: A critical review. Journal of Official Statistics, 29(3), 329–353. https://doi.org/10.2478/jos-2013-0026
Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3. https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2
Brown, T. (2008). Design thinking. Harvard Business Review, 86(6), 84–92.
Burke, P. (2000). A social history of knowledge: From Gutenberg to Diderot. Polity.
Card, D., & Krueger, A. B. (1994). Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania. American Economic Review, 84(4), 772–793.
Casey, E. (2011). Digital evidence and computer crime: Forensic science, computers and the internet (3rd ed.). Academic Press.
Chambers, C. D. (2013). Registered Reports: A new publishing initiative at Cortex. Cortex, 49(3), 609–610. https://doi.org/10.1016/j.cortex.2012.12.016
Chen, H., Chiang, R. H. L., & Storey, V. C. (2012). Business intelligence and analytics: From big data to big impact. MIS Quarterly, 36(4), 1165–1188. https://doi.org/10.2307/41703503
Christensen, C. M., Hall, T., Dillon, K., & Duncan, D. S. (2016). Know your customers' "jobs to be done." Harvard Business Review, 94(9), 54–62.
Coady, C. A. J. (1992). Testimony: A philosophical study. Clarendon Press.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
Cohen, W. M., & Levinthal, D. A. (1990). Absorptive capacity: A new perspective on learning and innovation. Administrative Science Quarterly, 35(1), 128–152. https://doi.org/10.2307/2393553
Converse, J. M. (1987). Survey research in the United States: Roots and emergence, 1890–1960. University of California Press.
Cooper, A. (1999). The inmates are running the asylum. Sams.
Cooperrider, D. L., & Srivastva, S. (1987). Appreciative inquiry in organizational life. Research in Organizational Change and Development, 1, 129–169.
Cressey, D. R. (1953). Other people's money: A study in the social psychology of embezzlement. Free Press.
Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). SAGE.
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555
Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993).
De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., Schroeder, T. V., Sox, H. C., & Van Der Weyden, M. B. (2004). Clinical trial registration: A statement from the International Committee of Medical Journal Editors. New England Journal of Medicine, 351(12), 1250–1251. https://doi.org/10.1056/NEJMe048225
de Leeuw, E. D. (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21(2), 233–255.
Denzin, N. K. (1978). The research act: A theoretical introduction to sociological methods (2nd ed.). McGraw-Hill.
Descartes, R. (1996). Meditations on first philosophy (J. Cottingham, Trans.; Rev. ed.). Cambridge University Press. (Original work published 1641)
Deville, J.-C., & Särndal, C.-E. (1992). Calibration estimators in survey sampling. Journal of the American Statistical Association, 87(418), 376–382. https://doi.org/10.1080/01621459.1992.10475217
Dewey, J. (1938). Logic: The theory of inquiry. Henry Holt.
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Wiley. https://doi.org/10.1002/9781394260645
Dixon, M., Freeman, K., & Toman, N. (2010). Stop trying to delight your customers. Harvard Business Review, 88(7/8), 116–122.
Dorminey, J., Fleming, A. S., Kranacher, M.-J., & Riley, R. A., Jr. (2012). The evolution of fraud theory. Issues in Accounting Education, 27(2), 555–579. https://doi.org/10.2308/iace-50131
Duncker, K. (1945). On problem-solving (L. S. Lees, Trans.). Psychological Monographs, 58(5), i–113. https://doi.org/10.1037/h0093599
Dwork, C., McSherry, F., Nissim, K., & Smith, A. (2006). Calibrating noise to sensitivity in private data analysis. In S. Halevi & T. Rabin (Eds.), Theory of cryptography (pp. 265–284). Springer. https://doi.org/10.1007/11681878_14
Edmondson, A. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44(2), 350–383. https://doi.org/10.2307/2666999
Edwards, W., Lindman, H., & Savage, L. J. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70(3), 193–242. https://doi.org/10.1037/h0044139
Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1–26. https://doi.org/10.1214/aos/1176344552
Eisenstein, E. L. (1979). The printing press as an agent of change. Cambridge University Press.
Elwert, F., & Winship, C. (2014). Endogenous selection bias: The problem of conditioning on a collider variable. Annual Review of Sociology, 40, 31–53. https://doi.org/10.1146/annurev-soc-071913-043455
Eppler, M. J., & Mengis, J. (2004). The concept of information overload: A review of literature from organization science, accounting, marketing, MIS, and related disciplines. The Information Society, 20(5), 325–344. https://doi.org/10.1080/01972240490507974
Etzkowitz, H., & Leydesdorff, L. (2000). The dynamics of innovation: From National Systems and "Mode 2" to a Triple Helix of university–industry–government relations. Research Policy, 29(2), 109–123. https://doi.org/10.1016/S0048-7333(99)00055-4
Facione, P. A. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction (The Delphi report). California Academic Press.
Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4(5), Article e5738. https://doi.org/10.1371/journal.pone.0005738
Financial Action Task Force. (2012). International standards on combating money laundering and the financing of terrorism & proliferation: The FATF recommendations. FATF/OECD.
Findley, K. A., & Scott, M. S. (2006). The multiple dimensions of tunnel vision in criminal cases. Wisconsin Law Review, 2006(2), 291–397.
Fisher, R. P., & Geiselman, R. E. (1992). Memory-enhancing techniques for investigative interviewing: The cognitive interview. Charles C Thomas.
Fishkin, J. S. (2009). When the people speak: Deliberative democracy and public consultation. Oxford University Press.
Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist, 34(10), 906–911. https://doi.org/10.1037/0003-066X.34.10.906
Floridi, L. (2005). Is semantic information meaningful data? Philosophy and Phenomenological Research, 70(2), 351–370. https://doi.org/10.1111/j.1933-1592.2005.tb00531.x
Foster, K. R., & Kokko, H. (2009). The evolution of superstitious and superstition-like behaviour. Proceedings of the Royal Society B: Biological Sciences, 276(1654), 31–37. https://doi.org/10.1098/rspb.2008.0981
Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484
Frické, M. (2009). The knowledge pyramid: A critique of the DIKW hierarchy. Journal of Information Science, 35(2), 131–142. https://doi.org/10.1177/0165551508094050
Fricker, M. (2007). Epistemic injustice: Power and the ethics of knowing. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780198237907.001.0001
Gadamer, H.-G. (2004). Truth and method (J. Weinsheimer & D. G. Marshall, Trans.; 2nd rev. ed.). Continuum. (Original work published 1960)
Galton, F. (1886). Regression towards mediocrity in hereditary stature. Journal of the Anthropological Institute of Great Britain and Ireland, 15, 246–263. https://doi.org/10.2307/2841583
Galton, F. (1907). Vox populi. Nature, 75(1949), 450–451. https://doi.org/10.1038/075450a0
Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2–3), 87–105. https://doi.org/10.1016/S1096-7516(00)00016-6
Geertz, C. (1973). The interpretation of cultures. Basic Books.
Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press. https://doi.org/10.1017/CBO9780511790942
Gettier, E. L. (1963). Is justified true belief knowledge? Analysis, 23(6), 121–123. https://doi.org/10.1093/analys/23.6.121
Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: The dynamics of science and research in contemporary societies. SAGE.
Gigerenzer, G., & Hoffrage, U. (1995). How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review, 102(4), 684–704. https://doi.org/10.1037/0033-295X.102.4.684
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine.
Glassman, M., & Kang, M. J. (2012). Intelligence in the internet age: The emergence and evolution of Open Source Intelligence (OSINT). Computers in Human Behavior, 28(2), 673–682. https://doi.org/10.1016/j.chb.2011.11.014
Goldman, A. I. (1999). Knowledge in a social world. Oxford University Press. https://doi.org/10.1093/0198238207.001.0001
Graves, L. (2016). Deciding what's true: The rise of political fact-checking in American journalism. Columbia University Press.
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. https://doi.org/10.1007/s10654-016-0149-3
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998). Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology, 74(6), 1464–1480. https://doi.org/10.1037/0022-3514.74.6.1464
Griffin, A., & Hauser, J. R. (1993). The voice of the customer. Marketing Science, 12(1), 1–27. https://doi.org/10.1287/mksc.12.1.1
Groves, R. M., & Couper, M. P. (1998). Nonresponse in household interview surveys. Wiley. https://doi.org/10.1002/9781118490082
Groves, R. M., & Heeringa, S. G. (2006). Responsive design for household surveys: Tools for actively controlling survey errors and costs. Journal of the Royal Statistical Society: Series A (Statistics in Society), 169(3), 439–457. https://doi.org/10.1111/j.1467-985X.2006.00423.x
Groves, R. M., & Lyberg, L. (2010). Total survey error: Past, present, and future. Public Opinion Quarterly, 74(5), 849–879. https://doi.org/10.1093/poq/nfq065
Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72(2), 167–189. https://doi.org/10.1093/poq/nfn011
Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70(5), 646–675. https://doi.org/10.1093/poq/nfl033
Groves, R. M., Fowler, F. J., Jr., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology (2nd ed.). Wiley.
Groves, R. M., Presser, S., & Dipko, S. (2004). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68(1), 2–31. https://doi.org/10.1093/poq/nfh002
Groves, R. M., Singer, E., & Corning, A. (2000). Leverage-saliency theory of survey participation. Public Opinion Quarterly, 64(3), 299–308. https://doi.org/10.1086/317990
Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), 59–82. https://doi.org/10.1177/1525822X05279903
Gupta, S., Hanssens, D., Hardie, B., Kahn, W., Kumar, V., Lin, N., Ravishanker, N., & Sriram, S. (2006). Modeling customer lifetime value. Journal of Service Research, 9(2), 139–155. https://doi.org/10.1177/1094670506293810
Hamill, R., Wilson, T. D., & Nisbett, R. E. (1980). Insensitivity to sample bias: Generalizing from atypical cases. Journal of Personality and Social Psychology, 39(4), 578–589. https://doi.org/10.1037/0022-3514.39.4.578
Harding, S. (1986). The science question in feminism. Cornell University Press.
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer. https://doi.org/10.1007/978-0-387-84858-7
Heckathorn, D. D. (1997). Respondent-driven sampling: A new approach to the study of hidden populations. Social Problems, 44(2), 174–199. https://doi.org/10.2307/3096941
Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004). A structural approach to selection bias. Epidemiology, 15(5), 615–625. https://doi.org/10.1097/01.ede.0000135174.63482.43
Heuer, R. J., Jr. (1999). Psychology of intelligence analysis. Center for the Study of Intelligence, Central Intelligence Agency.
Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235–266. https://doi.org/10.1023/B:EDPR.0000034022.16470.f3
Hogan, A., Blomqvist, E., Cochez, M., d'Amato, C., de Melo, G., Gutierrez, C., Kirrane, S., Labra Gayo, J. E., Navigli, R., Neumaier, S., Ngonga Ngomo, A.-C., Polleres, A., Rashid, S. M., Rula, A., Schmelzeisen, L., Sequeda, J., Staab, S., & Zimmermann, A. (2021). Knowledge graphs. ACM Computing Surveys, 54(4), Article 71. https://doi.org/10.1145/3447772
Hogan, C. E., Rezaee, Z., Riley, R. A., Jr., & Velury, U. K. (2008). Financial statement fraud: Insights from the academic literature. Auditing: A Journal of Practice & Theory, 27(2), 231–252. https://doi.org/10.2308/aud.2008.27.2.231
Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–960. https://doi.org/10.1080/01621459.1986.10478354
Horvitz, D. G., & Thompson, D. J. (1952). A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260), 663–685. https://doi.org/10.1080/01621459.1952.10483446
Howard, R. A. (1966). Information value theory. IEEE Transactions on Systems Science and Cybernetics, 2(1), 22–26. https://doi.org/10.1109/TSSC.1966.300074
Huber, G. P. (1991). Organizational learning: The contributing processes and the literatures. Organization Science, 2(1), 88–115. https://doi.org/10.1287/orsc.2.1.88
Hulnick, A. S. (2006). What's wrong with the intelligence cycle. Intelligence and National Security, 21(6), 959–979. https://doi.org/10.1080/02684520601046291
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), Article e124. https://doi.org/10.1371/journal.pmed.0020124
Irwin, D., & Mandel, D. R. (2019). Improving information evaluation for intelligence production. Intelligence and National Security, 34(4), 503–525. https://doi.org/10.1080/02684527.2019.1569343
Janis, I. L. (1972). Victims of groupthink: A psychological study of foreign-policy decisions and fiascoes. Houghton Mifflin.
Jaynes, E. T. (2003). Probability theory: The logic of science (G. L. Bretthorst, Ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511790423
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), Article 248. https://doi.org/10.1145/3571730
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953
Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80(4), 237–251. https://doi.org/10.1037/h0034747
Kahneman, D. (2011). Thinking, fast and slow. Farrar, Straus and Giroux.
Kahneman, D., Fredrickson, B. L., Schreiber, C. A., & Redelmeier, D. A. (1993). When more pain is preferred to less: Adding a better end. Psychological Science, 4(6), 401–405. https://doi.org/10.1111/j.1467-9280.1993.tb00589.x
Kalton, G., & Flores-Cervantes, I. (2003). Weighting methods. Journal of Official Statistics, 19(2), 81–98.
Kaplan, R. S., & Norton, D. P. (1992). The balanced scorecard: Measures that drive performance. Harvard Business Review, 70(1), 71–79.
Kapur, M. (2008). Productive failure. Cognition and Instruction, 26(3), 379–424. https://doi.org/10.1080/07370000802212669
Kassin, S. M., & Gudjonsson, G. H. (2004). The psychology of confessions: A review of the literature and issues. Psychological Science in the Public Interest, 5(2), 33–67. https://doi.org/10.1111/j.1529-1006.2004.00016.x
Kassin, S. M., Dror, I. E., & Kukucka, J. (2013). The forensic confirmation bias: Problems, perspectives, and proposed solutions. Journal of Applied Research in Memory and Cognition, 2(1), 42–52. https://doi.org/10.1016/j.jarmac.2013.01.001
Keiningham, T. L., Cooil, B., Andreassen, T. W., & Aksoy, L. (2007). A longitudinal examination of net promoter and firm revenue growth. Journal of Marketing, 71(3), 39–51. https://doi.org/10.1509/jmkg.71.3.039
Kent, S. (1949). Strategic intelligence for American world policy. Princeton University Press.
Kent, S. (1964). Words of estimative probability. Studies in Intelligence, 8(4), 49–65.
Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. https://doi.org/10.1207/s15327957pspr0203_4
King, G., Murray, C. J. L., Salomon, J. A., & Tandon, A. (2004). Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98(1), 191–207. https://doi.org/10.1017/S000305540400108X
Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75–86. https://doi.org/10.1207/s15326985ep4102_1
Kish, L. (1965). Survey sampling. Wiley.
Kish, L. (1992). Weighting for unequal Pi. Journal of Official Statistics, 8(2), 183–200.
Kitchin, R. (2014). The data revolution: Big data, open data, data infrastructures and their consequences. SAGE. https://doi.org/10.4135/9781473909472
Knaflic, C. N. (2015). Storytelling with data: A data visualization guide for business professionals. Wiley.
Knight, F. H. (1921). Risk, uncertainty and profit. Houghton Mifflin.
Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 140–181. https://doi.org/10.1007/s10618-008-0114-1
Kounios, J., & Beeman, M. (2014). The cognitive neuroscience of insight. Annual Review of Psychology, 65, 71–93. https://doi.org/10.1146/annurev-psych-010213-115154
Kovach, B., & Rosenstiel, T. (2014). The elements of journalism: What newspeople should know and the public should expect (3rd ed.). Three Rivers Press.
Kozinets, R. V. (2002). The field behind the screen: Using netnography for marketing research in online communities. Journal of Marketing Research, 39(1), 61–72. https://doi.org/10.1509/jmkr.39.1.61.18935
Kreuter, F. (Ed.). (2013). Improving surveys with paradata: Analytic uses of process information. Wiley. https://doi.org/10.1002/9781118596869
Krippendorff, K. (2019). Content analysis: An introduction to its methodology (4th ed.). SAGE.
Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236. https://doi.org/10.1002/acp.2350050305
Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537–567. https://doi.org/10.1146/annurev.psych.50.1.537
Kuhn, T. S. (1962). The structure of scientific revolutions. University of Chicago Press.
LaPiere, R. T. (1934). Attitudes vs. actions. Social Forces, 13(2), 230–237. https://doi.org/10.2307/2570339
Leary, M. R., Diebels, K. J., Davisson, E. K., Jongman-Sereno, K. P., Isherwood, J. C., Raimi, K. T., Deffler, S. A., & Hoyle, R. H. (2017). Cognitive and interpersonal features of intellectual humility. Personality and Social Psychology Bulletin, 43(6), 793–813. https://doi.org/10.1177/0146167217697695
Lemon, K. N., & Verhoef, P. C. (2016). Understanding customer experience throughout the customer journey. Journal of Marketing, 80(6), 69–96. https://doi.org/10.1509/jm.15.0420
Lewandowsky, S., Ecker, U. K. H., Seifert, C. M., Schwarz, N., & Cook, J. (2012). Misinformation and its correction: Continued influence and successful debiasing. Psychological Science in the Public Interest, 13(3), 106–131. https://doi.org/10.1177/1529100612451018
Lewin, K. (1946). Action research and minority problems. Journal of Social Issues, 2(4), 34–46. https://doi.org/10.1111/j.1540-4560.1946.tb02295.x
Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 22(140), 1–55.
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. SAGE.
Lipton, P. (2004). Inference to the best explanation (2nd ed.). Routledge.
Little, R. J. A., & Rubin, D. B. (2019). Statistical analysis with missing data (3rd ed.). Wiley. https://doi.org/10.1002/9781119482260
Loewenstein, G. (1994). The psychology of curiosity: A review and reinterpretation. Psychological Bulletin, 116(1), 75–98. https://doi.org/10.1037/0033-2909.116.1.75
Loftus, E. F., & Palmer, J. C. (1974). Reconstruction of automobile destruction: An example of the interaction between language and memory. Journal of Verbal Learning and Verbal Behavior, 13(5), 585–589. https://doi.org/10.1016/S0022-5371(74)80011-3
Lohr, S. L. (2021). Sampling: Design and analysis (3rd ed.). CRC Press. https://doi.org/10.1201/9780429298899
Luca, M., & Zervas, G. (2016). Fake it till you make it: Reputation, competition, and Yelp review fraud. Management Science, 62(12), 3412–3427. https://doi.org/10.1287/mnsc.2015.2304
Luhn, H. P. (1958). A business intelligence system. IBM Journal of Research and Development, 2(4), 314–319. https://doi.org/10.1147/rd.24.0314
Machlup, F. (1962). The production and distribution of knowledge in the United States. Princeton University Press.
Mandel, D. R., & Barnes, A. (2014). Accuracy of forecasts in strategic intelligence. Proceedings of the National Academy of Sciences, 111(30), 10984–10989. https://doi.org/10.1073/pnas.1406138111
Mangel, M., & Samaniego, F. J. (1984). Abraham Wald's work on aircraft survivability. Journal of the American Statistical Association, 79(386), 259–267. https://doi.org/10.1080/01621459.1984.10478038
McIntyre, L. (2018). Post-truth. MIT Press.
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085
Meadows, D. H. (2008). Thinking in systems: A primer. Chelsea Green.
Meiklejohn, S., Pomarole, M., Jordan, G., Levchenko, K., McCoy, D., Voelker, G. M., & Savage, S. (2013). A fistful of bitcoins: Characterizing payments among men with no names. In Proceedings of the 2013 Conference on Internet Measurement (pp. 127–140). ACM. https://doi.org/10.1145/2504730.2504747
Mellers, B., Stone, E., Murray, T., Minster, A., Rohrbaugh, N., Bishop, M., Chen, E., Baker, J., Hou, Y., Horowitz, M., Ungar, L., & Tetlock, P. (2015). Identifying and cultivating superforecasters as a method of improving probabilistic predictions. Perspectives on Psychological Science, 10(3), 267–281. https://doi.org/10.1177/1745691615577794
Memon, A., Meissner, C. A., & Fraser, J. (2010). The cognitive interview: A meta-analytic review and study space analysis of the past 25 years. Psychology, Public Policy, and Law, 16(4), 340–372. https://doi.org/10.1037/a0020518
Merton, R. K., & Barber, E. (2004). The travels and adventures of serendipity: A study in sociological semantics and the sociology of science. Princeton University Press.
Merton, R. K. (1965). On the shoulders of giants: A Shandean postscript. Free Press.
Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations (N. W. Storer, Ed.). University of Chicago Press.
Messerli, F. H. (2012). Chocolate consumption, cognitive function, and Nobel laureates. New England Journal of Medicine, 367(16), 1562–1564. https://doi.org/10.1056/NEJMon1211064
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741
Metropolis, N., & Ulam, S. (1949). The Monte Carlo method. Journal of the American Statistical Association, 44(247), 335–341. https://doi.org/10.1080/01621459.1949.10483310
Mick, D. G. (1986). Consumer research and semiotics: Exploring the morphology of signs, symbols, and significance. Journal of Consumer Research, 13(2), 196–213. https://doi.org/10.1086/209060
Milne, R., & Bull, R. (1999). Investigative interviewing: Psychology and practice. Wiley.
Moreau, L., & Missier, P. (Eds.). (2013). PROV-DM: The PROV data model (W3C Recommendation). World Wide Web Consortium. https://www.w3.org/TR/prov-dm/
Moustakas, C. (1990). Heuristic research: Design, methodology, and applications. SAGE.
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, Article 0021. https://doi.org/10.1038/s41562-016-0021
Murphy, A. H. (1973). A new vector partition of the probability score. Journal of Applied Meteorology, 12(4), 595–600. https://doi.org/10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2
National Research Council. (2009). Strengthening forensic science in the United States: A path forward. National Academies Press. https://doi.org/10.17226/12589
Near, J. P., & Miceli, M. P. (1985). Organizational dissidence: The case of whistle-blowing. Journal of Business Ethics, 4(1), 1–16. https://doi.org/10.1007/BF00382668
Nelder, J. A., & Wedderburn, R. W. M. (1972). Generalized linear models. Journal of the Royal Statistical Society: Series A (General), 135(3), 370–384. https://doi.org/10.2307/2344614
Neter, J., & Waksberg, J. (1964). A study of response errors in expenditures data from household interviews. Journal of the American Statistical Association, 59(305), 18–55. https://doi.org/10.1080/01621459.1964.10480699
Neyman, J. (1934). On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection. Journal of the Royal Statistical Society, 97(4), 558–625. https://doi.org/10.2307/2342192
Neyman, J. (1937). Outline of a theory of statistical estimation based on the classical theory of probability. Philosophical Transactions of the Royal Society of London. Series A, 236(767), 333–380. https://doi.org/10.1098/rsta.1937.0005
Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. https://doi.org/10.1037/1089-2680.2.2.175
Nonaka, I., & Takeuchi, H. (1995). The knowledge-creating company: How Japanese companies create the dynamics of innovation. Oxford University Press.
Nonaka, I. (1994). A dynamic theory of organizational knowledge creation. Organization Science, 5(1), 14–37. https://doi.org/10.1287/orsc.5.1.14
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114
Ohno, T. (1988). Toyota production system: Beyond large-scale production. Productivity Press.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872. https://doi.org/10.1016/j.jesp.2009.03.009
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, Article n71. https://doi.org/10.1136/bmj.n71
Pariser, E. (2011). The filter bubble: What the Internet is hiding from you. Penguin Press.
Park, D. K., Gelman, A., & Bafumi, J. (2004). Bayesian multilevel estimation with poststratification: State-level estimates from national polls. Political Analysis, 12(4), 375–385. https://doi.org/10.1093/pan/mph024
Paul, R., & Elder, L. (2006). The thinker's guide to the art of Socratic questioning. Foundation for Critical Thinking.
Pawson, R., & Tilley, N. (1997). Realistic evaluation. SAGE.
Pearl, J., & Mackenzie, D. (2018). The book of why: The new science of cause and effect. Basic Books.
Pearl, J. (2009). Causality: Models, reasoning, and inference (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511803161
Pearson, K. (1897). Mathematical contributions to the theory of evolution: On a form of spurious correlation which may arise when indices are used in the measurement of organs. Proceedings of the Royal Society of London, 60, 489–498. https://doi.org/10.1098/rspl.1896.0076
Pearson, K. (1900). On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 50(302), 157–175. https://doi.org/10.1080/14786440009463897
Pearson, K. (1901). On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11), 559–572. https://doi.org/10.1080/14786440109462720
Peirce, C. S. (1877). The fixation of belief. Popular Science Monthly, 12, 1–15.
Plassmann, H., Ramsøy, T. Z., & Milosavljevic, M. (2012). Branding the brain: A critical review and outlook. Journal of Consumer Psychology, 22(1), 18–36. https://doi.org/10.1016/j.jcps.2011.11.010
Polanyi, M. (1966). The tacit dimension. Doubleday.
Popper, K. R. (1959). The logic of scientific discovery. Hutchinson.
Porter, M. E. (1979). How competitive forces shape strategy. Harvard Business Review, 57(2), 137–145.
Powell, W. W., & Snellman, K. (2004). The knowledge economy. Annual Review of Sociology, 30, 199–220. https://doi.org/10.1146/annurev.soc.29.010202.100037
Ragin, C. C. (1987). The comparative method: Moving beyond qualitative and quantitative strategies. University of California Press.
Rao, J. N. K., & Molina, I. (2015). Small area estimation (2nd ed.). Wiley. https://doi.org/10.1002/9781118735855
Reason, J. (1990). Human error. Cambridge University Press. https://doi.org/10.1017/CBO9781139062367
Reason, J. (2000). Human error: Models and management. BMJ, 320(7237), 768–770. https://doi.org/10.1136/bmj.320.7237.768
Reichheld, F. F. (2003). The one number you need to grow. Harvard Business Review, 81(12), 46–54.
Ries, E. (2011). The lean startup. Crown Business.
Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4(2), 155–169. https://doi.org/10.1007/BF01405730
Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15(3), 351–357. https://doi.org/10.2307/2087176
Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press.
Roozenbeek, J., & van der Linden, S. (2019). Fake news game confers psychological resistance against online misinformation. Palgrave Communications, 5, Article 65. https://doi.org/10.1057/s41599-019-0279-9
Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638
Rothstein, D., & Santana, L. (2011). Make just one change: Teach students to ask their own questions. Harvard Education Press.
Rousseau, D. M. (2006). Is there such a thing as "evidence-based management"? Academy of Management Review, 31(2), 256–269. https://doi.org/10.5465/amr.2006.20208679
Rowley, J. (2007). The wisdom hierarchy: Representations of the DIKW hierarchy. Journal of Information Science, 33(2), 163–180. https://doi.org/10.1177/0165551506070706
Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701. https://doi.org/10.1037/h0037350
Ryle, G. (1949). The concept of mind. Hutchinson.
Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021). An excess of positive results: Comparing the standard psychology literature with Registered Reports. Advances in Methods and Practices in Psychological Science, 4(2). https://doi.org/10.1177/25152459211007467
Schoemaker, P. J. H. (1995). Scenario planning: A tool for strategic thinking. Sloan Management Review, 36(2), 25–40.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
Shapin, S. (1996). The scientific revolution. University of Chicago Press.
Shapiro, C., & Varian, H. R. (1999). Information rules: A strategic guide to the network economy. Harvard Business School Press.
Sheeran, P. (2002). Intention–behavior relations: A conceptual and empirical review. European Review of Social Psychology, 12(1), 1–36. https://doi.org/10.1080/14792772143000003
Shewhart, W. A. (1931). Economic control of quality of manufactured product. Van Nostrand.
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series B (Methodological), 13(2), 238–241. https://doi.org/10.1111/j.2517-6161.1951.tb00088.x
Sio, U. N., & Ormerod, T. C. (2009). Does incubation enhance problem solving? A meta-analytic review. Psychological Bulletin, 135(1), 94–120. https://doi.org/10.1037/a0014212
Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178–182. https://doi.org/10.1177/014107680609900414
Smith, W. R. (1956). Product differentiation and market segmentation as alternative marketing strategies. Journal of Marketing, 21(1), 3–8. https://doi.org/10.1177/002224295602100102
Snow, J. (1855). On the mode of communication of cholera (2nd ed.). John Churchill.
Sorensen, R. A. (1992). Thought experiments. Oxford University Press.
Sparrow, M. K. (1991). The application of network analysis to criminal intelligence: An assessment of the prospects. Social Networks, 13(3), 251–274. https://doi.org/10.1016/0378-8733(91)90008-H
Spearman, C. (1904). "General intelligence," objectively determined and measured. The American Journal of Psychology, 15(2), 201–292. https://doi.org/10.2307/1412107
Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103(2684), 677–680. https://doi.org/10.1126/science.103.2684.677
Student. (1908). The probable error of a mean. Biometrika, 6(1), 1–25. https://doi.org/10.1093/biomet/6.1.1
Surowiecki, J. (2004). The wisdom of crowds. Doubleday.
Tetlock, P. E., & Gardner, D. (2015). Superforecasting: The art and science of prediction. Crown.
Thaler, R. H., & Sunstein, C. R. (2008). Nudge: Improving decisions about health, wealth, and happiness. Yale University Press.
Thistlethwaite, D. L., & Campbell, D. T. (1960). Regression-discontinuity analysis: An alternative to the ex post facto experiment. Journal of Educational Psychology, 51(6), 309–317. https://doi.org/10.1037/h0044319
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267–288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x
Tourangeau, R., & Yan, T. (2007). Sensitive questions in surveys. Psychological Bulletin, 133(5), 859–883. https://doi.org/10.1037/0033-2909.133.5.859
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press. https://doi.org/10.1017/CBO9780511819322
Tufte, E. R. (1983). The visual display of quantitative information. Graphics Press.
Tukey, J. W. (1977). Exploratory data analysis. Addison-Wesley.
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131. https://doi.org/10.1126/science.185.4157.1124
Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211(4481), 453–458. https://doi.org/10.1126/science.7455683
Twardy, C. R. (2004). Argument maps improve critical thinking. Teaching Philosophy, 27(2), 95–116. https://doi.org/10.5840/teachphil200427213
Vosoughi, S., Roy, D., & Aral, S. (2018). The spread of true and false news online. Science, 359(6380), 1146–1151. https://doi.org/10.1126/science.aap9559
Vrij, A. (2008). Detecting lies and deceit: Pitfalls and opportunities (2nd ed.). Wiley.
Wallas, G. (1926). The art of thought. Harcourt, Brace.
Walsh, J. P., & Ungson, G. R. (1991). Organizational memory. Academy of Management Review, 16(1), 57–91. https://doi.org/10.5465/amr.1991.4278992
Walton, D. (2008). Informal logic: A pragmatic approach (2nd ed.). Cambridge University Press.
Wardle, C., & Derakhshan, H. (2017). Information disorder: Toward an interdisciplinary framework for research and policy making (Council of Europe report DGI(2017)09). Council of Europe.
Warner, S. L. (1965). Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association, 60(309), 63–69. https://doi.org/10.1080/01621459.1965.10480775
Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108
Weick, K. E. (1995). Sensemaking in organizations. SAGE.
Wells, G. L., & Olson, E. A. (2003). Eyewitness testimony. Annual Review of Psychology, 54, 277–295. https://doi.org/10.1146/annurev.psych.54.101601.145028
Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge University Press. https://doi.org/10.1017/CBO9780511803932
Wilensky, H. L. (1967). Organizational intelligence: Knowledge and policy in government and industry. Basic Books.
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., … Mons, B. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, Article 160018. https://doi.org/10.1038/sdata.2016.18
Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. SAGE. https://doi.org/10.4135/9781412983655
Wilson, A. M. (2001). Mystery shopping: Using deception to measure service performance. Psychology & Marketing, 18(7), 721–734. https://doi.org/10.1002/mar.1027
Wolfe, D. T., & Hermanson, D. R. (2004). The fraud diamond: Considering the four elements of fraud. The CPA Journal, 74(12), 38–42.
Wolfers, J., & Zitzewitz, E. (2004). Prediction markets. Journal of Economic Perspectives, 18(2), 107–126. https://doi.org/10.1257/0895330041371321
Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., & Malone, T. W. (2010). Evidence for a collective intelligence factor in the performance of human groups. Science, 330(6004), 686–688. https://doi.org/10.1126/science.1193147
Yin, R. K. (2018). Case study research and applications: Design and methods (6th ed.). SAGE.
Yule, G. U. (1926). Why do we sometimes get nonsense-correlations between time-series? A study in sampling and the nature of time-series. Journal of the Royal Statistical Society, 89(1), 1–63. https://doi.org/10.2307/2341482
Zenko, M. (2015). Red team: How to succeed by thinking like the enemy. Basic Books.
Zins, C. (2007). Conceptual approaches for defining data, information, and knowledge. Journal of the American Society for Information Science and Technology, 58(4), 479–493. https://doi.org/10.1002/asi.20508
The tendency to answer in ways that present oneself favourably, over-reporting approved behaviours such as voting and under-reporting disapproved ones such as drug use. It is strongest with interviewers present and on sensitive topics. Self-administration, assurances of confidentiality and indirect techniques such as randomized response reduce it. (Tourangeau & Yan, 2007)
Surveys · Nonresponse, weighting and data quality
See also: Randomized response technique, Mode effect, Say–do gap
Read more: Social Desirability Bias and Techniques for Reducing It · Asking Sensitive Questions with Care and Precision