Research

Research glossary

Terms for research philosophy, design, qualitative and quantitative methods, evidence synthesis and research integrity.

A

A/B test also: split test

A randomised controlled experiment, usually online, in which users are randomly assigned to two or more versions of a product, page or message and an outcome such as conversion is compared. Randomisation makes the versions comparable, so differences in outcome can be attributed to the change. Common pitfalls include stopping a test as soon as results look significant and testing many metrics without correction. (Kohavi et al., 2009)

Research design
See also: Randomised controlled trial, Multiple comparisons (Statistics), Statistical power (Statistics)
Read more: A/B Testing as Applied Experimental Research

Action research

An approach in which researchers and participants study a practical problem while acting to change it, moving through repeated cycles of planning, action, observation and reflection. Kurt Lewin coined the term in the 1940s. It values practical improvement and the knowledge of participants alongside formal findings. (Lewin, 1946)

Research design
See also: Case study
Read more: Action Research: Studying Change While Making It

C

Case study

An in-depth empirical study of a contemporary phenomenon in its real-life context, especially when the boundary between phenomenon and context is unclear. Case studies use multiple sources of evidence and can be single or multiple, exploratory, descriptive or explanatory. They generalise analytically, to theory, rather than statistically, to a population. (Yin, 2018)

Research design
See also: Triangulation
Read more: Case Study Research: Design, Rigour and Generalisation · Case Selection Strategies in Multiple-Case Research Designs

Coding

Labelling segments of qualitative data, such as interview passages, with short descriptive or interpretive codes so that they can be compared, grouped into categories and developed into themes or theory. Codes may come from the data (inductive) or from an existing framework (deductive). (Braun & Clarke, 2006; Glaser & Strauss, 1967)

Qualitative research
See also: Thematic analysis, Grounded theory, Content analysis
Read more: Coding Qualitative Data: Manual and Software-Assisted Approaches

Construct validity

The degree to which a measure actually captures the theoretical concept (construct) it is meant to measure. It is established by showing that scores relate to other measures as theory predicts: strongly with measures of the same construct (convergent evidence) and weakly with measures of different constructs (discriminant evidence). Modern validity theory treats it as the unifying form of validity. (Cronbach & Meehl, 1955; Messick, 1995)

Quantitative and mixed methods
See also: Reliability, Operationalisation, Internal validity
Read more: Construct Validity and How to Establish It · Reliability and Validity: The Twin Pillars of Measurement

Content analysis

A method for making replicable and valid inferences from texts, images or other communication by coding their content into categories according to explicit rules. Quantitative content analysis counts coded features; qualitative variants interpret them. Reliability is checked by having independent coders apply the same scheme. (Krippendorff, 2019)

Quantitative and mixed methods
See also: Thematic analysis, Inter-rater reliability (Statistics)
Read more: Content Analysis: Quantifying Text and Media

Critical realism

A philosophy of science holding that reality exists independently of our knowledge of it, but that our knowledge is fallible and socially produced. It distinguishes the real (structures and mechanisms), the actual (events) and the empirical (what is observed), and directs research to the mechanisms that produce outcomes in particular contexts. Realist evaluation draws on it. (Pawson & Tilley, 1997)

Philosophy and paradigms
See also: Realist evaluation, Positivism, Interpretivism
Read more: Critical Realist Approaches to Research and Stratified Knowledge Claims

E

Ethnography

The study of a group's culture and everyday practices through extended participant observation, interviews and documents, aiming to understand the world from members' perspectives. Clifford Geertz described its goal as "thick description": interpreting behaviour within the web of meanings that gives it significance. (Geertz, 1973)

Qualitative research
See also: Netnography
Read more: Ethnography and Participant Observation in Commercial Research · Ethnographic Inquiry and Understanding From the Inside

External validity

The extent to which a causal relationship found in a study holds across other people, settings, treatments and outcomes. Laboratory experiments can be strong on internal validity but weak on external validity; field experiments and replication across contexts strengthen it. (Shadish et al., 2002)

Quantitative and mixed methods
See also: Internal validity, Field experiment
Read more: Internal and External Validity in Research Design · Field Experiments Versus Laboratory Experiments

F

Falsifiability

Karl Popper's criterion for distinguishing scientific from non-scientific claims: a theory is scientific only if it forbids some possible observations, so that it could in principle be shown false by evidence. On this view science advances by bold conjectures and attempts at refutation rather than by accumulating confirmations. (Popper, 1959)

Philosophy and paradigms
See also: Hypothesis, Paradigm, Induction (Inquiry)
Read more: Karl Popper and Falsifiability as a Criterion of Science · The Scientific Method: Hypothesis, Test and Revision

Field experiment

A randomised experiment conducted in a natural setting, such as a workplace, market or website, where participants often do not know they are in a study. Field experiments trade some control for greater realism and therefore stronger external validity than laboratory experiments. (Shadish et al., 2002)

Research design
See also: Randomised controlled trial, A/B test, External validity
Read more: Field Experiments Versus Laboratory Experiments

File-drawer problem

The tendency for studies with null or non-significant results to remain unpublished, "in the file drawer", so that the published literature overstates the size and reliability of effects. Robert Rosenthal named the problem in 1979 and proposed estimating how many unpublished null studies would be needed to overturn a finding. (Rosenthal, 1979; Franco et al., 2014)

Research integrity
See also: Publication bias, Registered report, Meta-analysis
Read more: Publication Bias and the File-Drawer Problem

G

Grounded theory

A qualitative methodology, introduced by Barney Glaser and Anselm Strauss in 1967, for building theory from data rather than testing existing theory. It uses constant comparison between cases, open and focused coding, memo writing and theoretical sampling, continuing until new data no longer add to the emerging categories. (Glaser & Strauss, 1967)

Qualitative research
See also: Thematic analysis, Saturation, Coding
Read more: Grounded Theory: Building Theory From Data

H

HARKing

Hypothesising After the Results are Known: presenting a hypothesis formed after seeing the data as if it had been stated in advance. It makes exploratory findings look confirmatory and inflates false-positive rates. Pre-registration makes it detectable. (Kerr, 1998)

Research integrity
See also: P-hacking, Pre-registration, Questionable research practices
Read more: P-Hacking, HARKing and Questionable Research Practices

Hypothesis

A testable statement about the relationship between variables, derived from theory or prior observation, that research is designed to support or refute. A good hypothesis specifies the expected direction of an effect and could be shown false by the evidence. (Popper, 1959)

Research design
See also: Falsifiability, Operationalisation, Pre-registration
Read more: From Research Question to Hypothesis: Operationalising Concepts · The Scientific Method: Hypothesis, Test and Revision

I

Internal validity

The extent to which a study supports the conclusion that a relationship between variables is causal, rather than the product of confounding, selection, history, maturation or other threats. Random assignment is the strongest protection; quasi-experimental designs address the threats with design features and analysis. (Shadish et al., 2002)

Quantitative and mixed methods
See also: External validity, Confounding (Statistics), Quasi-experiment
Read more: Internal and External Validity in Research Design

Interpretivism

A philosophical position holding that social reality is constructed through the meanings people give to their actions, so that research must understand those meanings from the participants' point of view rather than only measure behaviour. It underpins much qualitative research, including ethnography and phenomenology. (Geertz, 1973; Lincoln & Guba, 1985)

Philosophy and paradigms
See also: Positivism, Ethnography, Hermeneutics (Inquiry)
Read more: Positivism, Interpretivism and Pragmatism in Research Design

M

Member checking also: respondent validation

Returning data, interpretations or findings to participants to ask whether they recognise them as accurate accounts of their experience. Lincoln and Guba described it as the most crucial technique for establishing credibility, though participants' views may legitimately differ from the researcher's analysis. (Lincoln & Guba, 1985)

Qualitative research
See also: Trustworthiness
Read more: The Use of Member Checking and Respondent Validation for Knowledge Confirmation

Meta-analysis

A statistical method for combining the results of multiple studies of the same question into a pooled estimate of effect size, weighting each study by its precision. Fixed-effect models assume one true effect; random-effects models allow the true effect to vary between studies. Heterogeneity statistics and checks for publication bias are standard parts of the method. (Borenstein et al., 2009)

Literature and evidence
See also: Systematic review, Effect size (Statistics), Publication bias
Read more: Meta-Analysis: Combining Results Across Studies · Meta-Regression Techniques for Explaining Heterogeneity in Aggregated Knowledge

Mixed methods research

Research that deliberately combines qualitative and quantitative data and analysis within one study or programme, and integrates the two. Core designs are convergent (both collected together and compared), explanatory sequential (qualitative work explains quantitative results) and exploratory sequential (qualitative work informs a later quantitative phase). (Creswell & Plano Clark, 2018)

Quantitative and mixed methods
See also: Triangulation, Thematic analysis
Read more: Mixed Methods Research: Convergent, Explanatory and Exploratory Designs

N

Natural experiment

A situation in which an external event, policy or rule assigns people to different conditions in a way that is as good as random, allowing causal effects to be studied without the researcher controlling assignment. Lotteries, arbitrary boundaries and sudden policy changes are common sources. (Angrist & Pischke, 2009; Card & Krueger, 1994)

Research design
See also: Quasi-experiment, Instrumental variable (Statistics), Difference-in-differences (Statistics)
Read more: Natural Experiments in Business and Social Research

Netnography

Ethnographic research adapted to online communities and social media, introduced by Robert Kozinets for marketing research. It studies the shared meanings and practices of online groups through observation, participation and analysis of their communications, with attention to consent and privacy. (Kozinets, 2002)

Qualitative research
See also: Ethnography
Read more: Netnography: Studying Online Communities · Online Qualitative Research: Communities, Forums and Asynchronous Groups

O

Open science

Practices that make research methods, data, materials, code and publications openly available so that others can check, reuse and build on them. It includes open access publishing, open data following the FAIR principles (findable, accessible, interoperable, reusable), pre-registration and shared analysis code. (Munafò et al., 2017; Wilkinson et al., 2016)

Research integrity
See also: Pre-registration, Replication crisis, FAIR data (The Generation of Knowledge)
Read more: Open Science: Open Data, Open Materials and Open Access · Research Data Management Plans and Long-Term Knowledge Preservation

Operationalisation

The process of turning an abstract concept into specific, observable measures or procedures, for example defining "customer loyalty" as repeat purchase within twelve months. Good operationalisation is explicit, so others can replicate the study, and valid, so the measure captures the concept. (Shadish et al., 2002)

Research design
See also: Construct validity, Hypothesis
Read more: From Research Question to Hypothesis: Operationalising Concepts

P

Paradigm

In Thomas Kuhn's account, the shared set of theories, methods, exemplary problems and standards that defines normal science in a field. Accumulating anomalies can lead to a crisis and a scientific revolution in which one paradigm replaces another. In social research the word is also used for broad positions such as positivism, interpretivism and pragmatism. (Kuhn, 1962)

Philosophy and paradigms
See also: Falsifiability, Positivism, Interpretivism
Read more: Thomas Kuhn, Paradigm Shifts and the Structure of Scientific Revolutions · Positivism, Interpretivism and Pragmatism in Research Design

Peer review

The evaluation of research by independent experts before publication or funding. It is meant to catch errors and judge importance, but studies show it is slow, inconsistent between reviewers, poor at detecting fraud, and open to bias. Open review, registered reports and post-publication review are attempts to improve it. (Smith, 2006)

Research integrity
See also: Registered report, Replication crisis
Read more: Peer Review: How It Works and How It Fails · Peer Review, Verification and Quality Control of Knowledge

P-hacking

Analysing data in many ways (adding or dropping observations, trying different covariates, outcomes or subgroups) and reporting the analyses that reach statistical significance. Even without intent to deceive, such flexibility can make false-positive findings very likely. (Simmons et al., 2011; John et al., 2012)

Research integrity
See also: HARKing, Pre-registration, P-value (Statistics)
Read more: P-Hacking, HARKing and Questionable Research Practices

Positivism

A philosophical position holding that genuine knowledge comes from observation and measurement, that the social world can be studied with the methods of the natural sciences, and that research should seek general, testable laws. Post-positivism keeps the commitment to empirical testing while accepting that all observation is fallible and theory-laden. (Popper, 1959; Kuhn, 1962)

Philosophy and paradigms
See also: Interpretivism, Critical realism, Paradigm
Read more: Positivism, Interpretivism and Pragmatism in Research Design · Ontology and Epistemology for Applied Researchers

Pre-registration also: preregistration

Recording a study's hypotheses, design and analysis plan in a time-stamped public registry before data are collected or analysed. It separates confirmatory from exploratory analysis and makes p-hacking and HARKing visible. Clinical trial registration has been required by major medical journals since 2005. (Nosek et al., 2018; De Angelis et al., 2004)

Research integrity
See also: Registered report, P-hacking, HARKing
Read more: Pre-Registration and Registered Reports

Publication bias

Systematic distortion of the published record that occurs when the chance of a study being published depends on the direction or significance of its results. It inflates average effect sizes in literature reviews and meta-analyses. Funnel plots and comparisons with registered studies are used to detect it. (Franco et al., 2014; Rosenthal, 1979)

Research integrity
See also: File-drawer problem, Meta-analysis, Registered report
Read more: Publication Bias and the File-Drawer Problem

Q

Qualitative comparative analysis also: QCA

A case-oriented method developed by Charles Ragin that uses Boolean algebra and set theory to identify which combinations of conditions are sufficient or necessary for an outcome across a small to medium number of cases. It is suited to causal complexity, where different combinations of conditions can lead to the same outcome. (Ragin, 1987)

Quantitative and mixed methods
See also: Case study
Read more: Configurational Comparative Methods (QCA) for Complex Causality Knowledge

Quasi-experiment

A study that tests a causal hypothesis by comparing outcomes under different conditions but without random assignment to those conditions. Designs such as non-equivalent control groups, interrupted time series and regression discontinuity use comparison groups, repeated measures and design features to rule out alternative explanations. (Shadish et al., 2002)

Research design
See also: Internal validity, Natural experiment, Regression discontinuity design (Statistics), Difference-in-differences (Statistics)
Read more: Quasi-Experimental Designs When Randomisation Is Not Possible

Questionable research practices also: QRPs

Practices that fall short of fabrication or falsification but inflate false-positive rates, such as selectively reporting outcomes, deciding to collect more data after looking at results, excluding data after seeing their effect, and presenting unexpected findings as predicted. A large survey of psychologists found such practices to be common. (John et al., 2012; Simmons et al., 2011)

Research integrity
See also: P-hacking, HARKing, Research misconduct
Read more: P-Hacking, HARKing and Questionable Research Practices

R

Randomised controlled trial also: RCT

An experiment in which participants are randomly allocated to an intervention or a control condition and outcomes are compared. Randomisation balances known and unknown confounders in expectation, which is why RCTs are regarded as the strongest design for estimating causal effects when they are feasible and ethical. (Shadish et al., 2002; Rubin, 1974)

Research design
See also: A/B test, Confounding (Statistics), Internal validity
Read more: Experimental Design: Randomisation, Control and Manipulation · Randomised Controlled Trials as the Benchmark for Causality

Realist evaluation

An approach to evaluation that asks what works, for whom, in what circumstances and why. It explains outcomes through context–mechanism–outcome configurations: programmes work by triggering mechanisms in people, and whether they do depends on context. (Pawson & Tilley, 1997)

Market and business research
See also: Critical realism
Read more: Realist Evaluation Approaches for Context-Mechanism-Outcome Knowledge · Program Evaluation Research as a Distinct Mode of Applied Knowledge Generation

Registered report

A journal article format in which the introduction, methods and analysis plan are peer reviewed before data are collected. If the protocol is accepted in principle, the journal commits to publishing the results whatever they show, which removes the incentive to produce positive findings. Registered reports report far fewer positive results than standard articles. (Chambers, 2013; Scheel et al., 2021)

Research integrity
See also: Pre-registration, Publication bias
Read more: Pre-Registration and Registered Reports

Reliability

The consistency of a measurement: the degree to which it gives the same result under the same conditions. Forms include test–retest reliability (stability over time), internal consistency (agreement among items, often measured by Cronbach's alpha) and inter-rater reliability (agreement among coders). A measure can be reliable without being valid. (Cronbach, 1951)

Quantitative and mixed methods
See also: Construct validity, Cronbach's alpha (Statistics), Inter-rater reliability (Statistics)
Read more: Reliability and Validity: The Twin Pillars of Measurement · Reliability Statistics: Cronbach's Alpha and Alternatives

Replication crisis

The finding, from large-scale replication projects in psychology, economics, medicine and other fields since about 2011, that many published results cannot be reproduced. In one project only about 36% of 100 psychology studies produced a significant result again. Causes include small samples, publication bias and flexible analysis; reforms include pre-registration and open data. (Open Science Collaboration, 2015; Ioannidis, 2005; Munafò et al., 2017)

Research integrity
See also: P-hacking, Publication bias, Pre-registration, Open science
Read more: The Replication Crisis: Causes, Evidence and Reforms · Research on Research (Meta-Research) for Improving Knowledge Generation Practices

Research misconduct also: FFP

Fabrication (making up data), falsification (manipulating data, materials or processes) or plagiarism (using others' work without credit) in proposing, performing, reviewing or reporting research. A meta-analysis of surveys found that about 2% of scientists admitted to having fabricated or falsified data at least once. (Fanelli, 2009)

Research integrity
See also: Questionable research practices, Peer review
Read more: Research Misconduct: Fabrication, Falsification and Plagiarism · Scientific Misconduct Investigations

S

Saturation

The point in qualitative data collection at which new interviews or observations no longer produce new themes or insights relevant to the research question. In one widely cited study, most themes appeared within the first twelve interviews in a fairly homogeneous group, though the number needed depends on the aims, population and analysis. (Guest et al., 2006; Glaser & Strauss, 1967)

Qualitative research
See also: Grounded theory, Thematic analysis
Read more: Saturation in Qualitative Research: How Many Interviews Are Enough?

Scoping review

A review that maps the extent, range and nature of research on a broad topic, identifies key concepts and gaps, and often precedes a systematic review. It follows a transparent search process but does not usually appraise study quality or pool results. (Arksey & O'Malley, 2005)

Literature and evidence
See also: Systematic review
Read more: Scoping Reviews and Rapid Evidence Assessments

Segmentation also: market segmentation

Dividing a heterogeneous market into smaller groups of customers with similar needs, characteristics or behaviour, so that products and marketing can be targeted to each. Wendell Smith introduced the concept in 1956. Segments are commonly derived from survey data with cluster or latent class analysis. (Smith, 1956)

Market and business research
See also: Cluster analysis (Statistics), Latent class analysis (Statistics), Persona (Insights)
Read more: Segmentation Research: Dividing Markets Into Meaningful Groups · Segmentation as a Source of Insight · Cluster Analysis for Market Segmentation

Systematic review

A review that answers a defined question by identifying, appraising and synthesising all relevant studies using explicit, reproducible methods: a protocol, comprehensive searches, predefined inclusion criteria and risk-of-bias assessment. The PRISMA 2020 statement sets out how such reviews should be reported. (Page et al., 2021)

Literature and evidence
See also: Meta-analysis, Scoping review
Read more: Systematic Reviews: Protocols, Searches and Synthesis

T

Thematic analysis

A method for identifying, analysing and reporting patterns of meaning (themes) in qualitative data. Braun and Clarke's widely used version has six phases: familiarisation, generating codes, constructing themes, reviewing themes, defining and naming themes, and writing up. (Braun & Clarke, 2006)

Qualitative research
See also: Grounded theory, Coding, Content analysis
Read more: Thematic Analysis: A Step-by-Step Method · Coding Qualitative Data: Manual and Software-Assisted Approaches

Triangulation

Using more than one method, data source, investigator or theory to study the same phenomenon, so that conclusions do not depend on the weaknesses of a single approach. Norman Denzin distinguished data, investigator, theory and methodological triangulation. (Denzin, 1978)

Quantitative and mixed methods
See also: Mixed methods research, Corroboration (Investigation)
Read more: Triangulation: Strengthening Conclusions With Multiple Methods

Trustworthiness

Lincoln and Guba's criteria for judging qualitative research, parallel to validity and reliability in quantitative work: credibility (confidence in the findings), transferability (applicability to other contexts), dependability (consistency of the process) and confirmability (grounding in the data rather than the researcher's bias). (Lincoln & Guba, 1985)

Qualitative research
See also: Saturation, Member checking, Reflexivity (Inquiry)
Read more: Trustworthiness in Qualitative Research: Credibility, Transferability, Dependability · The Use of Member Checking and Respondent Validation for Knowledge Confirmation

References

Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press. https://doi.org/10.1515/9781400829828

Arksey, H., & O'Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32. https://doi.org/10.1080/1364557032000119616

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Wiley. https://doi.org/10.1002/9780470743386

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Card, D., & Krueger, A. B. (1994). Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania. American Economic Review, 84(4), 772–793.

Chambers, C. D. (2013). Registered Reports: A new publishing initiative at Cortex. Cortex, 49(3), 609–610. https://doi.org/10.1016/j.cortex.2012.12.016

Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). SAGE.

Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555

De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., Schroeder, T. V., Sox, H. C., & Van Der Weyden, M. B. (2004). Clinical trial registration: A statement from the International Committee of Medical Journal Editors. New England Journal of Medicine, 351(12), 1250–1251. https://doi.org/10.1056/NEJMe048225

Denzin, N. K. (1978). The research act: A theoretical introduction to sociological methods (2nd ed.). McGraw-Hill.

Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4(5), Article e5738. https://doi.org/10.1371/journal.pone.0005738

Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484

Geertz, C. (1973). The interpretation of cultures. Basic Books.

Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine.

Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), 59–82. https://doi.org/10.1177/1525822X05279903

Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), Article e124. https://doi.org/10.1371/journal.pmed.0020124

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953

Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. https://doi.org/10.1207/s15327957pspr0203_4

Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 140–181. https://doi.org/10.1007/s10618-008-0114-1

Kozinets, R. V. (2002). The field behind the screen: Using netnography for marketing research in online communities. Journal of Marketing Research, 39(1), 61–72. https://doi.org/10.1509/jmkr.39.1.61.18935

Krippendorff, K. (2019). Content analysis: An introduction to its methodology (4th ed.). SAGE.

Kuhn, T. S. (1962). The structure of scientific revolutions. University of Chicago Press.

Lewin, K. (1946). Action research and minority problems. Journal of Social Issues, 2(4), 34–46. https://doi.org/10.1111/j.1540-4560.1946.tb02295.x

Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. SAGE.

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741

Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, Article 0021. https://doi.org/10.1038/s41562-016-0021

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, Article n71. https://doi.org/10.1136/bmj.n71

Pawson, R., & Tilley, N. (1997). Realistic evaluation. SAGE.

Popper, K. R. (1959). The logic of scientific discovery. Hutchinson.

Ragin, C. C. (1987). The comparative method: Moving beyond qualitative and quantitative strategies. University of California Press.

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638

Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701. https://doi.org/10.1037/h0037350

Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021). An excess of positive results: Comparing the standard psychology literature with Registered Reports. Advances in Methods and Practices in Psychological Science, 4(2). https://doi.org/10.1177/25152459211007467

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632

Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178–182. https://doi.org/10.1177/014107680609900414

Smith, W. R. (1956). Product differentiation and market segmentation as alternative marketing strategies. Journal of Marketing, 21(1), 3–8. https://doi.org/10.1177/002224295602100102

Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., … Mons, B. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, Article 160018. https://doi.org/10.1038/sdata.2016.18

Yin, R. K. (2018). Case study research and applications: Design and methods (6th ed.). SAGE.