Skip to content

Investigation · Law, ethics and regulation

Ethical Boundaries in Investigative Research

Research that investigates people's behaviour, institutions or hidden practices can generate valuable knowledge, but it can also deceive, expose or harm the people involved. This article explains the core principles of research ethics, how they apply to covert, field and online investigation, and how researchers and businesses can decide where the boundaries lie.

ModQu research article · 2,238 words · 24 references (APA 7th edition)

Download PDF

Investigative research seeks to find out what people and organisations actually do, as opposed to what they say they do. A researcher may send matched job applications to employers to test for discrimination, join a group to observe it from the inside, analyse large volumes of social media posts, or survey people about sensitive behaviour. Such research can produce knowledge that no other method could, and much of it serves the public interest. It can also deceive people, intrude on their privacy, expose them to risk or treat them merely as sources of data.

Research ethics is the set of principles and procedures that help researchers decide when such methods are justified and how to reduce the harm they cause. This article explains where those principles came from, what they require, how they apply to the most difficult forms of investigative research, and how researchers, businesses and survey organisations can apply them in practice.

How research ethics developed

Modern research ethics grew out of scandals. After the Second World War, the trials of Nazi doctors who had performed experiments on prisoners led to the Nuremberg Code of 1947, which established that voluntary consent is essential. The World Medical Association's Declaration of Helsinki, first adopted in 1964 and revised many times since, set out ethical principles for medical research involving human participants (World Medical Association, 2013).

In the United States, the Tuskegee study, in which the Public Health Service observed the progress of untreated syphilis in hundreds of Black men from 1932 to 1972 without telling them their diagnosis or offering them treatment after penicillin became available, caused public outrage when it was revealed. Brandt (1978) showed how racism shaped both the design of the study and its continuation for forty years. The scandal led to the creation of a national commission whose Belmont Report (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979) set out the principles that still underpin research ethics regulation in the United States, codified in federal regulations known as the Common Rule (45 C.F.R. § 46, 2018). Other countries developed similar frameworks, such as Canada's Tri-Council Policy Statement (Canadian Institutes of Health Research et al., 2022).

Social science had its own controversies. Milgram (1963) led participants to believe that they were giving increasingly severe electric shocks to another person, in order to study obedience to authority; many participants showed extreme stress. Baumrind (1964) criticised the study for deceiving participants and causing them distress without adequate justification or protection. Humphreys (1970) observed men having sex in public toilets by acting as a lookout, then traced their car registration numbers to find their addresses and later interviewed them at home in disguise, under the pretext of a health survey. His research produced important findings about a hidden population but exposed participants to grave risk had their identities been revealed. Rosenhan (1973) reported that healthy volunteers who feigned symptoms had been admitted to psychiatric hospitals and then struggled to be released; the study influenced psychiatric diagnosis, but a later investigation by Cahalan (2019) raised serious doubts about whether parts of it had been conducted and reported as described, illustrating that investigative research must itself be open to investigation.

The core principles

The Belmont Report identified three principles, which have been adopted, with variations, in most research ethics frameworks.

Table of three principles: respect for persons, beneficence and justice, with what each requires and the typical tension it creates in investigative research.
Figure 1. The three Belmont principles and the tensions they create for investigative research. Ethical review asks whether the research's value justifies any departure from these principles. Diagram by ModQu.

Respect for persons requires that people be treated as autonomous agents who decide for themselves whether to take part in research. In practice, this means informed consent: participants should understand what the research involves, its risks and benefits, and their right to withdraw. People with reduced autonomy, such as children or people with certain cognitive impairments, need additional protection.

Beneficence requires that researchers do not harm participants and that they maximise possible benefits and minimise possible harms. Harms include physical injury but also psychological distress, embarrassment, loss of reputation, legal jeopardy and loss of privacy. Protecting the confidentiality of what participants reveal is one of the main ways researchers reduce harm.

Justice requires that the burdens and benefits of research be distributed fairly, so that some groups are not used as research subjects for the benefit of others. Tuskegee was a flagrant violation of this principle.

Professional codes translate these principles into specific rules for particular disciplines, such as the American Psychological Association's (2017) code of conduct for psychologists, the British Sociological Association's (2017) statement of ethical practice and the international code for market, opinion and social research maintained by ESOMAR (2016).

Deception and covert research

Deception and covert observation are the most contested methods in investigative research. They are also sometimes the only way to study behaviour that people would conceal or change if they knew they were being observed.

Horizontal scale from more open to more covert with five points: full disclosure, partial disclosure, deception with debriefing, field experiment and covert participation.
Figure 2. A spectrum of openness. Most ethics codes allow movement to the right only when the research has significant value, no less deceptive method would work and the risks to participants are low. Diagram by ModQu.

Most codes permit deception only when the research has significant scientific, educational or social value, no effective non-deceptive alternative is available, and the deception is not expected to cause serious harm, and they usually require that participants be told the truth afterwards (American Psychological Association [APA], 2017). Bulmer (1982) argued that covert participant observation is rarely justified, because alternatives usually exist and because covert research damages the trust on which social research depends. Calvey (2017), in contrast, argues that covert research has a long and valuable history, from studies of total institutions to studies of violent groups, and that a blanket rejection of it would leave important areas of social life unstudied.

Field experiments are a widely used form of investigative research in which people are unaware that they are in a study. Audit studies of discrimination are a leading example. Bertrand and Mullainathan (2004) sent thousands of fictitious job applications to employers in Boston and Chicago, identical except for the names, and found that applications with names typical of white applicants received 50 per cent more callbacks than those with names typical of Black applicants. Pager (2003) sent matched pairs of real testers to apply for jobs, varying whether they reported a criminal record, and found strong effects of both race and a criminal record on callbacks. Such studies deceive employers and take a small amount of their time, but they produce evidence of discrimination that is very hard to obtain otherwise. Most ethics committees have accepted them, on the grounds that the harm to each employer is small, that the research serves an important public interest, and that the method requires that employers not be told.

Research in digital environments

Online and data-intensive research has created new ethical questions. In 2014, researchers published a study in which Facebook had altered the news feeds of almost 700,000 users to show more or fewer positive or negative posts, in order to test whether emotions spread through social networks (Kramer et al., 2014). The study was legal under Facebook's terms of service, but it caused widespread criticism because users had not given meaningful consent to having their emotions experimentally manipulated, and the journal published an editorial expression of concern (Verma, 2014).

Zimmer (2010) examined a study that collected Facebook profile data from a cohort of university students and released a dataset that, although stripped of names, allowed students to be identified. He challenged the argument that research using data "already public" raises no ethical issues, noting that users had expectations about who would see their information. The Association of Internet Researchers' ethical guidelines emphasise that the ethics of online research depend on context, including the expectations of users, the sensitivity of the data and the vulnerability of the people concerned (franzke et al., 2020). Salganik (2018) argues that digital-age research requires researchers to apply principles rather than rules, because existing regulations do not anticipate many of the situations they face, and adds respect for law and the public interest to the Belmont principles.

Investigating the powerful

Some ethical frameworks were designed to protect vulnerable participants in medical research, and they can fit poorly with research that investigates powerful organisations. Requiring informed consent from a company before studying whether it discriminates, or from a government agency before examining its misconduct, would allow the subjects of the research to block it. Israel (2015) notes that social scientists have argued for a different balance when the research subject is an institution exercising power, rather than an individual who might be harmed. Haggerty (2004) warned of "ethics creep", in which review processes designed for biomedical research expand to cover social research in ways that discourage critical study of institutions without protecting anyone.

Even so, investigating institutions involves individuals: employees, managers and officials whose careers and reputations may be affected. Researchers should consider how to protect individuals who are not responsible for the practices studied, for example by reporting findings at the level of the organisation rather than naming junior staff.

Commercial and survey research

The same principles apply to research carried out by businesses and survey organisations, which is often less formally reviewed than academic research. The international code for market and social research sets out duties that parallel the academic principles: researchers must be honest with participants about who they are and what the research is for, obtain consent for the collection and use of personal data, allow people to refuse or withdraw, protect participants' identities, and never present selling or marketing as research (ESOMAR, 2016).

Several issues arise particularly often in commercial work. Online surveys may recruit respondents through websites, advertising or panels, so researchers should make clear at the start who is conducting the survey, how long it will take, what will happen to the answers and whether the results will be linked to other information about the respondent. Incentives should be fair but not so large that they pressure people to take part or to answer untruthfully. Questions about sensitive matters, such as health, finances or sexual behaviour, should be optional and should explain why they are being asked. Results reported to clients should be aggregated so that individuals cannot be identified, and raw data should be retained only as long as necessary.

Commercial research can also use investigative methods such as mystery shopping, in which researchers pose as customers to assess service. Professional guidelines generally accept this, because it assesses the organisation rather than exposing individuals, but they require that the results not be used to discipline named staff unless staff have been told that mystery shopping may take place. Applying ethical standards consistently is not only a legal and moral obligation; it is also what makes respondents willing to take part in future research, on which the whole industry depends.

Practical steps

Researchers, survey organisations and businesses conducting investigative research can apply the principles through a series of practical questions:

A short example

The following hypothetical case illustrates these steps. A research company wants to know whether estate agents in a city treat prospective tenants differently depending on their ethnicity. It considers a survey of tenants, but recognises that people often cannot know whether they were treated differently. It therefore designs an audit study: pairs of trained testers, matched in age, gender, stated income and manner, but differing in ethnicity, will make enquiries about the same advertised properties.

The company submits the design to an independent ethics review. The reviewers accept that the agents cannot be told in advance, because that would change their behaviour. They require that each agent's time be kept to a short enquiry with no viewings booked, that no agent or agency be named in the results, that the testers be trained and supported in case of hostile treatment, and that the findings be shared with the regulator and agents' professional body. The study finds that testers from minority groups were offered fewer viewings, and the professional body introduces training and monitoring for its members.

Conclusion

Investigative research can reveal hidden discrimination, misconduct and behaviour that people would not willingly disclose. Its methods, including deception, covert observation, field experiments and the analysis of online data, raise real ethical tensions with the principles of respect for persons, beneficence and justice. Those tensions are not resolved by banning such methods or by ignoring the principles, but by weighing the value of the knowledge against the harm, choosing the least intrusive effective method, protecting the people involved, seeking independent review and reporting honestly. Research that meets these standards earns the trust on which all future research depends.

References

American Psychological Association. (2017). Ethical principles of psychologists and code of conduct (2002, amended effective June 1, 2010, and January 1, 2017).

Baumrind, D. (1964). Some thoughts on ethics of research: After reading Milgram's "Behavioral study of obedience." American Psychologist, 19(6), 421–423. https://doi.org/10.1037/h0040128

Bertrand, M., & Mullainathan, S. (2004). Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. American Economic Review, 94(4), 991–1013. https://doi.org/10.1257/0002828042002561

Brandt, A. M. (1978). Racism and research: The case of the Tuskegee Syphilis Study. The Hastings Center Report, 8(6), 21–29. https://doi.org/10.2307/3561468

British Sociological Association. (2017). Statement of ethical practice.

Bulmer, M. (1982). When is disguise justified? Alternatives to covert participant observation. Qualitative Sociology, 5(4), 251–264. https://doi.org/10.1007/BF00986753

Cahalan, S. (2019). The great pretender: The undercover mission that changed our understanding of madness. Grand Central Publishing.

Calvey, D. (2017). Covert research: The art, politics and ethics of undercover fieldwork. SAGE.

Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, & Social Sciences and Humanities Research Council of Canada. (2022). Tri-Council Policy Statement: Ethical conduct for research involving humans – TCPS 2 (2022).

ESOMAR. (2016). ICC/ESOMAR international code on market, opinion and social research and data analytics.

franzke, a. s., Bechmann, A., Zimmer, M., Ess, C., & the Association of Internet Researchers. (2020). Internet research: Ethical guidelines 3.0. Association of Internet Researchers.

Haggerty, K. D. (2004). Ethics creep: Governing social science research in the name of ethics. Qualitative Sociology, 27(4), 391–414. https://doi.org/10.1023/B:QUAS.0000049239.15922.a3

Humphreys, L. (1970). Tearoom trade: Impersonal sex in public places. Aldine.

Israel, M. (2015). Research ethics and integrity for social scientists: Beyond regulatory compliance (2nd ed.). SAGE.

Kramer, A. D. I., Guillory, J. E., & Hancock, J. T. (2014). Experimental evidence of massive-scale emotional contagion through social networks. Proceedings of the National Academy of Sciences, 111(24), 8788–8790. https://doi.org/10.1073/pnas.1320040111

Milgram, S. (1963). Behavioral study of obedience. The Journal of Abnormal and Social Psychology, 67(4), 371–378. https://doi.org/10.1037/h0040525

National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. (1979). The Belmont report: Ethical principles and guidelines for the protection of human subjects of research. U.S. Department of Health, Education, and Welfare.

Pager, D. (2003). The mark of a criminal record. American Journal of Sociology, 108(5), 937–975. https://doi.org/10.1086/374403

Protection of Human Subjects, 45 C.F.R. § 46 (2018).

Rosenhan, D. L. (1973). On being sane in insane places. Science, 179(4070), 250–258. https://doi.org/10.1126/science.179.4070.250

Salganik, M. J. (2018). Bit by bit: Social research in the digital age. Princeton University Press.

Verma, I. M. (2014). Editorial expression of concern: Experimental evidence of massive-scale emotional contagion through social networks. Proceedings of the National Academy of Sciences, 111(29), 10779. https://doi.org/10.1073/pnas.1412469111

World Medical Association. (2013). World Medical Association Declaration of Helsinki: Ethical principles for medical research involving human subjects. JAMA, 310(20), 2191–2194. https://doi.org/10.1001/jama.2013.281053

Zimmer, M. (2010). "But the data is already public": On the ethics of research in Facebook. Ethics and Information Technology, 12(4), 313–325. https://doi.org/10.1007/s10676-010-9227-5