AutoRISE: Agent-Driven Strategy Evolution for Red-Teaming Large Language Models

Automated red-teaming for large language models (LLMs) typically optimizes attack prompts within a fixed, human-designed strategy, leaving the attack strategy itself unchanged. A new paper from researchers introduces AutoRISE, a method that instead optimizes the strategy by searching over executable attack programs rather than individual prompts. At each iteration, a coding agent edits a strategy, and a fixed evaluation harness scores the resulting attacks, returning both a scalar objective and per-example diagnostics that guide subsequent edits. This allows structural changes, including new attack components and altered control flow, that prompt-level methods cannot directly express.

AutoRISE was evaluated on 11 models from five families against seven established jailbreak datasets, using two benchmark suites developed on disjoint target sets. Across held-out models, it improved average attack success rate by 17.0 points over the strongest baseline, and up to 16 points on frontier targets with low baseline success rates. Ablations suggest these gains arise from unrestricted program search, particularly compositional techniques and control-flow edits. Notably, AutoRISE operates in a black-box, inference-only setting, requiring no fine-tuning, human annotation, or GPU compute, making it highly accessible for security researchers.