Investigation · Method and the future
Quality Assurance and Peer Review Processes for Investigative Findings
Investigative findings can be wrong in ways the original investigator cannot see. This article explains how quality assurance and peer review catch errors before findings are relied on, the levels of review from self-checking to independent verification, techniques that make review effective, and how to build review into an investigation.
Investigators are human, and their findings can be wrong. They may miss evidence, misread a document, accept an unreliable source, overlook an alternative explanation or let early beliefs shape later judgements. Because these errors are usually invisible to the person who makes them, investigations need processes that bring a second pair of eyes to the work before findings are relied on. Quality assurance covers the systems that make good work likely, such as standards, training and procedures. Peer review is the specific practice of having another qualified person examine the findings and the evidence behind them. This article explains why both matter, the forms they take and how to make them effective.
Why review is needed
Errors in investigation often arise from cognitive bias rather than carelessness. Confirmation bias leads people to seek and weigh evidence in ways that support what they already believe (Nickerson, 1998). Rossmo (2009) found that tunnel vision and related biases were common causes of serious investigative failures. Kassin et al. (2013) showed that forensic judgements, often assumed to be objective, can be shaped by contextual information such as knowledge of a confession.
The misidentification of Brandon Mayfield's fingerprint in connection with the 2004 Madrid train bombings shows the limits of review that is not independent. Several FBI examiners confirmed the initial identification, but the review found that later examiners knew the earlier conclusion and were influenced by it (Office of the Inspector General, 2006). Review is most effective when the reviewer forms an independent judgement.
Review also works. A large study of fingerprint examiners found that false positive identifications were rare and that most errors would have been caught by blind verification by a second examiner, because different examiners rarely made the same mistake (Ulery et al., 2011).
Levels of review
Self-checking. Checklists help investigators confirm that they have covered the scope, recorded sources, considered alternatives and supported every finding with evidence.
Supervisory review. Supervisors check that the investigation followed procedure and that its conclusions follow from the evidence. Professional standards for investigations in United States federal inspector general offices require supervisory review and an internal quality assurance programme (Council of the Inspectors General on Integrity and Efficiency [CIGIE], 2011).
Independent peer review. A qualified person who was not part of the investigation examines the evidence and reasoning. Peer review in science has well-known weaknesses, including inconsistency and bias (Smith, 2006), but a structured review of a specific investigation, with access to the evidence, can catch errors the author cannot see.
Blind verification. For key analytical findings, such as a forensic comparison or a calculation, a second examiner repeats the analysis without knowing the first result. Laboratory standards require procedures to assure the validity of results, which often include such checks (International Organization for Standardization [ISO], 2017). Linear sequential unmasking limits what case information an examiner sees before forming a judgement (Dror et al., 2015).
External audit. Periodic review of a sample of closed cases against standards identifies systemic problems that case-by-case review misses. Accredited inspection and testing bodies are audited in this way (ISO, 2012).
Proportionate review
Not every finding needs the same level of review. A minor complaint resolved informally may need only a self-check and a supervisor's signature, while a finding that could end someone's career, lead to a prosecution or be published needs independent review of the evidence. A simple approach is to grade investigations by the seriousness of possible outcomes and the complexity of the evidence, and to set the minimum level of review for each grade in advance. This prevents review from being skipped under time pressure in the cases that need it most, and avoids wasting reviewers' time on routine matters.
What reviewers should check
A reviewer's checklist typically covers:
- Scope. Were all the questions in the mandate answered, and nothing outside it?
- Evidence. Is every finding supported by evidence cited in the report, and was the evidence obtained and handled properly?
- Contrary evidence. Was evidence pointing the other way identified and addressed?
- Alternatives. Were other explanations considered and tested?
- Standard. Was the correct standard of proof applied, and stated?
- Fairness. Were the people affected told the allegations and given a chance to respond?
- Calculations and analysis. Can key figures be reproduced from the source data?
- Language. Are findings stated neutrally, without overstatement?
Reviewers should record what they checked, what they found and how any disagreement was resolved. That record is itself part of the investigation file, and it shows later readers that the findings were tested before they were relied on.
Making review effective
Review works best when it is structured. Reviewers should have a clear brief: to check that each finding is supported, that evidence against it has been addressed, that alternative explanations were considered and that the standard of proof was correctly applied. Structured analytic techniques can help. Devil's advocacy and red-team reviews deliberately argue against the conclusions to test them, and analysis of competing hypotheses checks whether evidence really discriminates between explanations (Heuer & Pherson, 2015). A "premortem", in which the team imagines that the finding later proved wrong and asks why, can surface weaknesses before they matter (Klein, 2007). Intelligence analytic standards require analysts to consider alternatives and express uncertainty, giving reviewers explicit criteria to check against (Office of the Director of National Intelligence [ODNI], 2015).
Organisational culture matters too. Review should be routine and expected, not a sign of distrust, and reviewers must feel able to challenge senior investigators. Disagreements and their resolution should be recorded.
A short example
The following hypothetical case shows review at work. An investigator concludes that a supplier inflated invoices. Before the report is finalised, a peer reviewer from another team examines the evidence and notices that the price comparison used list prices from a later year, after a general price rise. Recalculating with prices from the correct year reduces the apparent overcharge from 18 per cent to 4 per cent, within the contract's permitted variation for urgent orders. The finding is revised, the method error is recorded, and the team adds a check of comparison dates to its review checklist.
Conclusion
Quality assurance and peer review protect investigative findings from errors that investigators cannot see in their own work. Review ranges from self-checking to independent verification and external audit, and its value depends on independence, structure and a culture that welcomes challenge. Building proportionate review into every investigation is one of the most effective ways to make findings reliable.
References
Council of the Inspectors General on Integrity and Efficiency. (2011). Quality standards for investigations.
Dror, I. E., Thompson, W. C., Meissner, C. A., Kornfield, I., Krane, D., Saks, M., & Risinger, M. (2015). Context management toolbox: A linear sequential unmasking (LSU) approach for minimizing cognitive bias in forensic decision making. Journal of Forensic Sciences, 60(4), 1111–1112. https://doi.org/10.1111/1556-4029.12805
Heuer, R. J., Jr., & Pherson, R. H. (2015). Structured analytic techniques for intelligence analysis (2nd ed.). CQ Press.
International Organization for Standardization. (2012). Conformity assessment — Requirements for the operation of various types of bodies performing inspection (ISO/IEC 17020:2012).
International Organization for Standardization. (2017). General requirements for the competence of testing and calibration laboratories (ISO/IEC 17025:2017).
Kassin, S. M., Dror, I. E., & Kukucka, J. (2013). The forensic confirmation bias: Problems, perspectives, and proposed solutions. Journal of Applied Research in Memory and Cognition, 2(1), 42–52. https://doi.org/10.1016/j.jarmac.2013.01.001
Klein, G. (2007). Performing a project premortem. Harvard Business Review, 85(9), 18–19.
Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. https://doi.org/10.1037/1089-2680.2.2.175
Office of the Director of National Intelligence. (2015). Intelligence Community Directive 203: Analytic standards.
Office of the Inspector General. (2006). A review of the FBI's handling of the Brandon Mayfield case. U.S. Department of Justice.
Rossmo, D. K. (2009). Criminal investigative failures. CRC Press.
Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178–182. https://doi.org/10.1177/014107680609900414
Ulery, B. T., Hicklin, R. A., Buscaglia, J., & Roberts, M. A. (2011). Accuracy and reliability of forensic latent fingerprint decisions. Proceedings of the National Academy of Sciences, 108(19), 7733–7738. https://doi.org/10.1073/pnas.1018707108