LLMs cut static analysis alert fatigue with 98% recall in new study

Static analysis tools are essential for finding security weaknesses before deployment, but their high false-alarm rate overwhelms human reviewers. A new preprint from Klieber, Svoboda, Flynn, and Martins (arXiv:2607.09979) tests whether large language models can reliably classify alerts as real bugs or false alarms. They evaluated several reasoning LLMs—including o4-mini, gpt-oss-120b, and gpt-oss-20b—on three standard test suites. Two mistake-mitigation methods were applied: a consistency check (CC) that runs the LLM multiple times and requires unanimous verdicts, and an LLM reasoning evaluation (LRE) step that aggregates reasoning across runs before asking for a final decision. Across all three suites, the best setups yielded at least 98% recall and 94.8% specificity, meaning nearly all real bugs were flagged and almost all false alarms were correctly dismissed.

The work goes beyond simple classification. To guard against potential memorization in the Juliet suite, the authors based their generalization claims primarily on FormAI, which they manually adjudicated. They also introduced a novel validity check: they prompted the LLM to synthesize a program (a “trigger driver”) that dynamically reproduces the alleged flaw. Every trigger driver generated for an actual false alarm failed a validity check, while successful triggers strongly confirmed real vulnerabilities. This provides independent, reproducible evidence that an alert deserves attention. The paper demonstrates that current mid-tier reasoning models, combined with lightweight error-reduction techniques, can automate alert triage with near-perfect accuracy—offering a practical path to dramatically reduce the manual review burden in software security pipelines.