LLM screening errors decoded: 3 failure modes and fixes

Systematic literature reviews are critical but tedious, requiring experts to screen thousands of paper titles and abstracts. LLMs promise to automate this, but reliability remains a concern. A new study by Mika Mäntylä and colleagues (accepted at SEAA 2026) goes beyond simple accuracy metrics to qualitatively investigate LLM failures. They examined six software engineering systematic reviews spanning over 1,000 primary study papers. Two LLMs and human experts independently screened papers in zero-shot mode. Agreement ranged from Cohen's Kappa 0.52 to 0.77—moderate to substantial. But the researchers wanted to understand the 'why' behind disagreements. Through qualitative analysis of discrepancies, they identified three distinct categories of errors. The zero-shot setup, without any fine-tuning, highlights the baseline performance of general-purpose LLMs, making the findings broadly applicable.

The first failure mode is boundary ambiguity in key terms—LLMs misinterpreted the scope of inclusion criteria. The second is keyword overemphasis, where the model latched onto specific words while ignoring context. The third is incorrect topic inference, leading to irrelevant papers being included. Based on these findings, the team proposes actionable recommendations: validate the LLM's semantic understanding before deployment, run multiple LLMs in parallel and compare results, and focus validation efforts on borderline cases where disagreement is most likely. The authors call for community efforts to develop normative guidelines for LLM use in systematic reviews. This work provides a framework for improving LLM-assisted screening, moving from black-box acceptance to transparent, reliable automation that can reduce manual effort while maintaining quality.