Research
Research glossary
Terms for research philosophy, design, qualitative and quantitative methods, evidence synthesis and research integrity.
A
- A/B test also: split test
A randomised controlled experiment, usually online, in which users are randomly assigned to two or more versions of a product, page or message and an outcome such as conversion is compared. Randomisation makes the versions comparable, so differences in outcome can be attributed to the change. Common pitfalls include stopping a test as soon as results look significant and testing many metrics without correction. (Kohavi et al., 2009)
- Action research
An approach in which researchers and participants study a practical problem while acting to change it, moving through repeated cycles of planning, action, observation and reflection. Kurt Lewin coined the term in the 1940s. It values practical improvement and the knowledge of participants alongside formal findings. (Lewin, 1946)
C
- Case study
An in-depth empirical study of a contemporary phenomenon in its real-life context, especially when the boundary between phenomenon and context is unclear. Case studies use multiple sources of evidence and can be single or multiple, exploratory, descriptive or explanatory. They generalise analytically, to theory, rather than statistically, to a population. (Yin, 2018)
- Coding
Labelling segments of qualitative data, such as interview passages, with short descriptive or interpretive codes so that they can be compared, grouped into categories and developed into themes or theory. Codes may come from the data (inductive) or from an existing framework (deductive). (Braun & Clarke, 2006; Glaser & Strauss, 1967)
- Construct validity
The degree to which a measure actually captures the theoretical concept (construct) it is meant to measure. It is established by showing that scores relate to other measures as theory predicts: strongly with measures of the same construct (convergent evidence) and weakly with measures of different constructs (discriminant evidence). Modern validity theory treats it as the unifying form of validity. (Cronbach & Meehl, 1955; Messick, 1995)
- Content analysis
A method for making replicable and valid inferences from texts, images or other communication by coding their content into categories according to explicit rules. Quantitative content analysis counts coded features; qualitative variants interpret them. Reliability is checked by having independent coders apply the same scheme. (Krippendorff, 2019)
- Critical realism
A philosophy of science holding that reality exists independently of our knowledge of it, but that our knowledge is fallible and socially produced. It distinguishes the real (structures and mechanisms), the actual (events) and the empirical (what is observed), and directs research to the mechanisms that produce outcomes in particular contexts. Realist evaluation draws on it. (Pawson & Tilley, 1997)
E
- Ethnography
The study of a group's culture and everyday practices through extended participant observation, interviews and documents, aiming to understand the world from members' perspectives. Clifford Geertz described its goal as "thick description": interpreting behaviour within the web of meanings that gives it significance. (Geertz, 1973)
- External validity
The extent to which a causal relationship found in a study holds across other people, settings, treatments and outcomes. Laboratory experiments can be strong on internal validity but weak on external validity; field experiments and replication across contexts strengthen it. (Shadish et al., 2002)
F
- Falsifiability
Karl Popper's criterion for distinguishing scientific from non-scientific claims: a theory is scientific only if it forbids some possible observations, so that it could in principle be shown false by evidence. On this view science advances by bold conjectures and attempts at refutation rather than by accumulating confirmations. (Popper, 1959)
- Field experiment
A randomised experiment conducted in a natural setting, such as a workplace, market or website, where participants often do not know they are in a study. Field experiments trade some control for greater realism and therefore stronger external validity than laboratory experiments. (Shadish et al., 2002)
- File-drawer problem
The tendency for studies with null or non-significant results to remain unpublished, "in the file drawer", so that the published literature overstates the size and reliability of effects. Robert Rosenthal named the problem in 1979 and proposed estimating how many unpublished null studies would be needed to overturn a finding. (Rosenthal, 1979; Franco et al., 2014)
G
- Grounded theory
A qualitative methodology, introduced by Barney Glaser and Anselm Strauss in 1967, for building theory from data rather than testing existing theory. It uses constant comparison between cases, open and focused coding, memo writing and theoretical sampling, continuing until new data no longer add to the emerging categories. (Glaser & Strauss, 1967)
H
- HARKing
Hypothesising After the Results are Known: presenting a hypothesis formed after seeing the data as if it had been stated in advance. It makes exploratory findings look confirmatory and inflates false-positive rates. Pre-registration makes it detectable. (Kerr, 1998)
- Hypothesis
A testable statement about the relationship between variables, derived from theory or prior observation, that research is designed to support or refute. A good hypothesis specifies the expected direction of an effect and could be shown false by the evidence. (Popper, 1959)
I
- Internal validity
The extent to which a study supports the conclusion that a relationship between variables is causal, rather than the product of confounding, selection, history, maturation or other threats. Random assignment is the strongest protection; quasi-experimental designs address the threats with design features and analysis. (Shadish et al., 2002)
- Interpretivism
A philosophical position holding that social reality is constructed through the meanings people give to their actions, so that research must understand those meanings from the participants' point of view rather than only measure behaviour. It underpins much qualitative research, including ethnography and phenomenology. (Geertz, 1973; Lincoln & Guba, 1985)
M
- Member checking also: respondent validation
Returning data, interpretations or findings to participants to ask whether they recognise them as accurate accounts of their experience. Lincoln and Guba described it as the most crucial technique for establishing credibility, though participants' views may legitimately differ from the researcher's analysis. (Lincoln & Guba, 1985)
- Meta-analysis
A statistical method for combining the results of multiple studies of the same question into a pooled estimate of effect size, weighting each study by its precision. Fixed-effect models assume one true effect; random-effects models allow the true effect to vary between studies. Heterogeneity statistics and checks for publication bias are standard parts of the method. (Borenstein et al., 2009)
- Mixed methods research
Research that deliberately combines qualitative and quantitative data and analysis within one study or programme, and integrates the two. Core designs are convergent (both collected together and compared), explanatory sequential (qualitative work explains quantitative results) and exploratory sequential (qualitative work informs a later quantitative phase). (Creswell & Plano Clark, 2018)
N
- Natural experiment
A situation in which an external event, policy or rule assigns people to different conditions in a way that is as good as random, allowing causal effects to be studied without the researcher controlling assignment. Lotteries, arbitrary boundaries and sudden policy changes are common sources. (Angrist & Pischke, 2009; Card & Krueger, 1994)
- Netnography
Ethnographic research adapted to online communities and social media, introduced by Robert Kozinets for marketing research. It studies the shared meanings and practices of online groups through observation, participation and analysis of their communications, with attention to consent and privacy. (Kozinets, 2002)
O
- Open science
Practices that make research methods, data, materials, code and publications openly available so that others can check, reuse and build on them. It includes open access publishing, open data following the FAIR principles (findable, accessible, interoperable, reusable), pre-registration and shared analysis code. (Munafò et al., 2017; Wilkinson et al., 2016)
- Operationalisation
The process of turning an abstract concept into specific, observable measures or procedures, for example defining "customer loyalty" as repeat purchase within twelve months. Good operationalisation is explicit, so others can replicate the study, and valid, so the measure captures the concept. (Shadish et al., 2002)
P
- Paradigm
In Thomas Kuhn's account, the shared set of theories, methods, exemplary problems and standards that defines normal science in a field. Accumulating anomalies can lead to a crisis and a scientific revolution in which one paradigm replaces another. In social research the word is also used for broad positions such as positivism, interpretivism and pragmatism. (Kuhn, 1962)
- Peer review
The evaluation of research by independent experts before publication or funding. It is meant to catch errors and judge importance, but studies show it is slow, inconsistent between reviewers, poor at detecting fraud, and open to bias. Open review, registered reports and post-publication review are attempts to improve it. (Smith, 2006)
- P-hacking
Analysing data in many ways (adding or dropping observations, trying different covariates, outcomes or subgroups) and reporting the analyses that reach statistical significance. Even without intent to deceive, such flexibility can make false-positive findings very likely. (Simmons et al., 2011; John et al., 2012)
- Positivism
A philosophical position holding that genuine knowledge comes from observation and measurement, that the social world can be studied with the methods of the natural sciences, and that research should seek general, testable laws. Post-positivism keeps the commitment to empirical testing while accepting that all observation is fallible and theory-laden. (Popper, 1959; Kuhn, 1962)
- Pre-registration also: preregistration
Recording a study's hypotheses, design and analysis plan in a time-stamped public registry before data are collected or analysed. It separates confirmatory from exploratory analysis and makes p-hacking and HARKing visible. Clinical trial registration has been required by major medical journals since 2005. (Nosek et al., 2018; De Angelis et al., 2004)
- Publication bias
Systematic distortion of the published record that occurs when the chance of a study being published depends on the direction or significance of its results. It inflates average effect sizes in literature reviews and meta-analyses. Funnel plots and comparisons with registered studies are used to detect it. (Franco et al., 2014; Rosenthal, 1979)
Q
- Qualitative comparative analysis also: QCA
A case-oriented method developed by Charles Ragin that uses Boolean algebra and set theory to identify which combinations of conditions are sufficient or necessary for an outcome across a small to medium number of cases. It is suited to causal complexity, where different combinations of conditions can lead to the same outcome. (Ragin, 1987)
- Quasi-experiment
A study that tests a causal hypothesis by comparing outcomes under different conditions but without random assignment to those conditions. Designs such as non-equivalent control groups, interrupted time series and regression discontinuity use comparison groups, repeated measures and design features to rule out alternative explanations. (Shadish et al., 2002)
- Questionable research practices also: QRPs
Practices that fall short of fabrication or falsification but inflate false-positive rates, such as selectively reporting outcomes, deciding to collect more data after looking at results, excluding data after seeing their effect, and presenting unexpected findings as predicted. A large survey of psychologists found such practices to be common. (John et al., 2012; Simmons et al., 2011)
R
- Randomised controlled trial also: RCT
An experiment in which participants are randomly allocated to an intervention or a control condition and outcomes are compared. Randomisation balances known and unknown confounders in expectation, which is why RCTs are regarded as the strongest design for estimating causal effects when they are feasible and ethical. (Shadish et al., 2002; Rubin, 1974)
- Realist evaluation
An approach to evaluation that asks what works, for whom, in what circumstances and why. It explains outcomes through context–mechanism–outcome configurations: programmes work by triggering mechanisms in people, and whether they do depends on context. (Pawson & Tilley, 1997)
- Registered report
A journal article format in which the introduction, methods and analysis plan are peer reviewed before data are collected. If the protocol is accepted in principle, the journal commits to publishing the results whatever they show, which removes the incentive to produce positive findings. Registered reports report far fewer positive results than standard articles. (Chambers, 2013; Scheel et al., 2021)
- Reliability
The consistency of a measurement: the degree to which it gives the same result under the same conditions. Forms include test–retest reliability (stability over time), internal consistency (agreement among items, often measured by Cronbach's alpha) and inter-rater reliability (agreement among coders). A measure can be reliable without being valid. (Cronbach, 1951)
- Replication crisis
The finding, from large-scale replication projects in psychology, economics, medicine and other fields since about 2011, that many published results cannot be reproduced. In one project only about 36% of 100 psychology studies produced a significant result again. Causes include small samples, publication bias and flexible analysis; reforms include pre-registration and open data. (Open Science Collaboration, 2015; Ioannidis, 2005; Munafò et al., 2017)
- Research misconduct also: FFP
Fabrication (making up data), falsification (manipulating data, materials or processes) or plagiarism (using others' work without credit) in proposing, performing, reviewing or reporting research. A meta-analysis of surveys found that about 2% of scientists admitted to having fabricated or falsified data at least once. (Fanelli, 2009)
S
- Saturation
The point in qualitative data collection at which new interviews or observations no longer produce new themes or insights relevant to the research question. In one widely cited study, most themes appeared within the first twelve interviews in a fairly homogeneous group, though the number needed depends on the aims, population and analysis. (Guest et al., 2006; Glaser & Strauss, 1967)
- Scoping review
A review that maps the extent, range and nature of research on a broad topic, identifies key concepts and gaps, and often precedes a systematic review. It follows a transparent search process but does not usually appraise study quality or pool results. (Arksey & O'Malley, 2005)
- Segmentation also: market segmentation
Dividing a heterogeneous market into smaller groups of customers with similar needs, characteristics or behaviour, so that products and marketing can be targeted to each. Wendell Smith introduced the concept in 1956. Segments are commonly derived from survey data with cluster or latent class analysis. (Smith, 1956)
- Systematic review
A review that answers a defined question by identifying, appraising and synthesising all relevant studies using explicit, reproducible methods: a protocol, comprehensive searches, predefined inclusion criteria and risk-of-bias assessment. The PRISMA 2020 statement sets out how such reviews should be reported. (Page et al., 2021)
T
- Thematic analysis
A method for identifying, analysing and reporting patterns of meaning (themes) in qualitative data. Braun and Clarke's widely used version has six phases: familiarisation, generating codes, constructing themes, reviewing themes, defining and naming themes, and writing up. (Braun & Clarke, 2006)
- Triangulation
Using more than one method, data source, investigator or theory to study the same phenomenon, so that conclusions do not depend on the weaknesses of a single approach. Norman Denzin distinguished data, investigator, theory and methodological triangulation. (Denzin, 1978)
- Trustworthiness
Lincoln and Guba's criteria for judging qualitative research, parallel to validity and reliability in quantitative work: credibility (confidence in the findings), transferability (applicability to other contexts), dependability (consistency of the process) and confirmability (grounding in the data rather than the researcher's bias). (Lincoln & Guba, 1985)
References
Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press. https://doi.org/10.1515/9781400829828
Arksey, H., & O'Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32. https://doi.org/10.1080/1364557032000119616
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Wiley. https://doi.org/10.1002/9780470743386
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Card, D., & Krueger, A. B. (1994). Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania. American Economic Review, 84(4), 772–793.
Chambers, C. D. (2013). Registered Reports: A new publishing initiative at Cortex. Cortex, 49(3), 609–610. https://doi.org/10.1016/j.cortex.2012.12.016
Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). SAGE.
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555
De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., Schroeder, T. V., Sox, H. C., & Van Der Weyden, M. B. (2004). Clinical trial registration: A statement from the International Committee of Medical Journal Editors. New England Journal of Medicine, 351(12), 1250–1251. https://doi.org/10.1056/NEJMe048225
Denzin, N. K. (1978). The research act: A theoretical introduction to sociological methods (2nd ed.). McGraw-Hill.
Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4(5), Article e5738. https://doi.org/10.1371/journal.pone.0005738
Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484
Geertz, C. (1973). The interpretation of cultures. Basic Books.
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine.
Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), 59–82. https://doi.org/10.1177/1525822X05279903
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), Article e124. https://doi.org/10.1371/journal.pmed.0020124
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953
Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. https://doi.org/10.1207/s15327957pspr0203_4
Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 140–181. https://doi.org/10.1007/s10618-008-0114-1
Kozinets, R. V. (2002). The field behind the screen: Using netnography for marketing research in online communities. Journal of Marketing Research, 39(1), 61–72. https://doi.org/10.1509/jmkr.39.1.61.18935
Krippendorff, K. (2019). Content analysis: An introduction to its methodology (4th ed.). SAGE.
Kuhn, T. S. (1962). The structure of scientific revolutions. University of Chicago Press.
Lewin, K. (1946). Action research and minority problems. Journal of Social Issues, 2(4), 34–46. https://doi.org/10.1111/j.1540-4560.1946.tb02295.x
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. SAGE.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, Article 0021. https://doi.org/10.1038/s41562-016-0021
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, Article n71. https://doi.org/10.1136/bmj.n71
Pawson, R., & Tilley, N. (1997). Realistic evaluation. SAGE.
Popper, K. R. (1959). The logic of scientific discovery. Hutchinson.
Ragin, C. C. (1987). The comparative method: Moving beyond qualitative and quantitative strategies. University of California Press.
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638
Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701. https://doi.org/10.1037/h0037350
Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021). An excess of positive results: Comparing the standard psychology literature with Registered Reports. Advances in Methods and Practices in Psychological Science, 4(2). https://doi.org/10.1177/25152459211007467
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178–182. https://doi.org/10.1177/014107680609900414
Smith, W. R. (1956). Product differentiation and market segmentation as alternative marketing strategies. Journal of Marketing, 21(1), 3–8. https://doi.org/10.1177/002224295602100102
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., … Mons, B. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, Article 160018. https://doi.org/10.1038/sdata.2016.18
Yin, R. K. (2018). Case study research and applications: Design and methods (6th ed.). SAGE.