Detect--Repair--Verify for LLM-Generated Code: A Multi-Language, Multi-Granularity Empirical Study

A new empirical study by researcher Cheng Cheng tackles the critical challenge of securing code generated by Large Language Models (LLMs). The research introduces a comprehensive Detect-Repair-Verify (DRV) workflow and a novel benchmark called 'EduCollab' to systematically evaluate the security of AI-generated software artifacts. EduCollab is a multi-language, multi-granularity benchmark consisting of runnable web applications in PHP, JavaScript, and Python, each paired with executable functional and security exploit test suites. This addresses a significant gap in the field: the lack of test-grounded benchmarks for end-to-end evaluation of LLM-generated code.

The study compares three approaches: unrepaired baselines, single-pass detect-repair, and bounded iterative DRV cycles, all under comparable computational budget constraints. The key metric is 'secure-and-correct yield'—the percentage of applications that pass both security and functional tests. Results show that bounded iterative DRV can improve this yield over single-pass repair, but the gains are uneven. Improvements are more pronounced at narrower repair scopes (like file-level) than at the complex project level. The research also reveals that automated vulnerability detection reports are often useful for guiding repairs, but their reliability is inconsistent, and the trustworthiness of the final repaired code depends heavily on the scope of the repair task.

These findings underscore that securing AI-generated code is not a one-step process. They highlight the necessity for robust, iterative workflows and rigorous, test-based verification to build trustworthy software with LLMs. The study provides crucial evidence for developers and security teams implementing AI coding assistants, pushing the industry toward more reliable and secure AI-powered software engineering practices.