LLMs evolve from script kiddies to autonomous pen-testing agents via RLVR

A comprehensive survey by Zheyuan He, Jiaxun Dong, and seven co-authors (arXiv:2607.02605) systematically reviews 81 papers published between 2023 and 2026 on LLM-driven penetration testing—a field they term Agents4Pentest. The authors propose a unified taxonomy that categorizes the literature into six groups: evaluation benchmarks, general-purpose systems, domain-specific frameworks, CTF-based systems, defense-oriented research, and surveys. They trace a four-phase architectural evolution: text-only reasoning agents, tool-augmented agents, memory-augmented agents, and finally agents trained with Reinforcement Learning with Verifiable Rewards (RLVR). Each transition was driven by a distinct capability bottleneck—e.g., RLVR shifts from imitating expert demonstrations to reward-driven self-improvement, enabling agents to discover previously undocumented attack strategies.

The survey yields several critical findings. CTF platforms have evolved from mere evaluation testbeds into dual-purpose infrastructure for both agent evaluation and RL training. Domain-specific frameworks improve efficiency through recurring specialization mechanisms, but their gains are largely confined to narrow task classes and are difficult to compare across domains due to reliance on different benchmarks. The field is also expanding beyond offensive automation toward adversarial defense and security compliance. Despite progress, the authors identify three structurally linked open challenges: evaluation reliability (how to ensure benchmark scores reflect real-world skill), limited performance on multi-stage attack scenarios, and scarcity of high-quality training data. The paper provides a roadmap for researchers and practitioners aiming to build more capable and trustworthy autonomous penetration testing systems.