Study: LLMs are often targets, not root cause, in software vulnerabilities

A new paper from researchers at multiple institutions introduces LLMCVE, the first dedicated dataset for LLM-in-the-loop vulnerabilities. The team collected 2,888 multi-source vulnerabilities across 230 popular LLM components (e.g., frameworks, APIs) and manually filtered them to 205 that strictly fit the definition: vulnerabilities introduced by the LLM or its downstream dependencies. Their analysis reveals a surprising insight: LLMs most often act as targets of attacks or as vectors that propagate existing flaws, rather than being the root cause themselves. This reframes the security narrative around AI-integrated software.

To assess real-world impact, the researchers tested SWE-Agent, an automated vulnerability repair system, on LLMCVE. Results show that LLM-in-the-loop vulnerabilities are significantly harder to fix than conventional software bugs. For prompt injection attacks—where malicious input forces the LLM to override its instructions—the Pass@1 rate was just 28.57%. Other vulnerability types fared better but still lagged behind typical software repairs. The study underscores that current agent-based repair methods are ill-equipped for the nuanced, context-dependent flaws unique to LLM-integrated systems, calling for new approaches in AI security.