Fuzzing with Agents? Generators Are All You Need
LLM agents automatically create input generators that make traditional coverage-guided mutation obsolete.
A new research paper titled "Fuzzing with Agents? Generators Are All You Need" introduces Gentoo, a system where an LLM coding agent is tasked with automatically writing specialized input generators for software testing (fuzzing). The core hypothesis challenges a long-standing practice: instead of using lightweight, generic generators and relying heavily on random mutation guided by code coverage, can an AI agent create a single, smart generator that understands the target program's structure so well that mutation becomes unnecessary? The Gentoo agent is given terminal access and the source code of the target library, then instructed to iteratively write and refine a Python generator.
The researchers evaluated three configurations of Gentoo against human-written generators on fuzz targets for 7 real-world Java libraries. The results were striking: agent-synthesized generators achieved statistically higher branch coverage than the human baselines on 4 out of 7 benchmarks. The most significant finding, however, was that coverage guidance and mutation—a cornerstone of modern fuzzing—provided no statistically significant benefit for the AI-generated fuzzers. This stands in direct contrast to the human-written generators, which still relied on these techniques. The implication is that the LLM agents are successfully encoding deep structural and semantic logic about the target, creating generators that are inherently more effective at exploring complex program states without needing random, coverage-driven nudges.
This work, available on arXiv (ID: 2604.01442), points toward a future where the heavy lifting of creating domain-specific, intelligent test harnesses could be automated. It shifts the developer's role from writing complex, error-prone generators to defining the problem for an AI agent and validating its output. While focused on Java libraries, the methodology suggests a broader applicability for automating software quality assurance tasks that traditionally require deep, manual expertise.
- Gentoo's AI agents beat human-written fuzzers, achieving higher branch coverage on 4 out of 7 real-world Java library benchmarks.
- The AI-generated fuzzers made coverage-guided mutation statistically unnecessary, a technique critical for all human-written generators in the study.
- The system gives an LLM agent terminal access and source code to iteratively synthesize and refine a target-specific Python input generator.
Why It Matters
Automates the creation of high-quality software tests, potentially saving developer time and uncovering more bugs through smarter, AI-generated logic.