AI models fail to detect vulnerability fixes from code alone, study finds

A new comprehensive study from researchers at the University of Lübeck has cast doubt on the ability of code language models to autonomously detect vulnerability-fixing commits (VFCs) using only code changes. The paper, titled "Code-Centric Detection of Vulnerability-Fixing Commits: A Unified Benchmark and Empirical Study," consolidates over 20 fragmented datasets spanning more than 180,000 commits into a unified framework. The team fine-tuned models ranging from 125 million to 14 billion parameters across more than 180 experiments, systematically testing whether code-centric approaches could reliably identify security patches.

The results are sobering. The models showed no evidence of acquiring transferable security-relevant code understanding from code changes alone. When commit messages were available, they dominated model attention, and removing them caused attribution analysis to show that enriching diffs with additional intra-procedural semantic context did not shift attention toward the code changes. Group-stratified evaluation revealed approximately 17% performance drops compared to random splits, while temporal splits on aggregated datasets proved unreliable due to compositional shift. At a false positive rate of 0.5%, all fine-tuned code-only models missed over 93% of vulnerabilities. Larger models and more diverse training data showed only preliminary improvements without resolving underlying limitations. The researchers have released their unified framework and evaluation suite to help advance code-centric VFC detection.