Fletcher & Stevenson's confidence-based stopping cuts review screening by 40%
New heuristic methods decide to stop when enough info is gathered, not just recall.
A new paper from Aaron Fletcher and Mark Stevenson tackles a critical bottleneck in evidence-based medicine: the immense manual effort required for systematic reviews. Current Technology Assisted Review (TAR) stopping methods typically focus on achieving a target recall rate—ensuring a certain percentage of relevant documents are found. This ignores whether the information need has actually been satisfied, often leading to over-screening.
The authors propose two heuristic stopping methods that monitor screened documents for sufficiency of information to reach a conclusion. Evaluated on standard Diagnostic Test Accuracy Systematic Review datasets, the methods substantially reduce the number of documents that need examination while, in the majority of cases, maintaining conclusions that are consistent with the full evidence set. This could cut weeks of work for researchers and accelerate meta-analyses without sacrificing rigor.
- Two new heuristic stopping methods shift focus from recall targets to information sufficiency.
- Tested on Diagnostic Test Accuracy Systematic Reviews—reduces document screening substantially.
- Conclusions remain consistent with all evidence in the majority of cases despite fewer documents.
Why It Matters
Saves researchers weeks of manual screening, accelerating systematic reviews without compromising accuracy.