Refactor Arena: A Control Setting for Software Engineering

Researchers fastfedora and Tyler Tracy have released Refactor Arena, an open-source platform designed to test AI safety in realistic software engineering scenarios. Unlike previous benchmarks like APPS that used simple functions, Refactor Arena simulates complex codebases where AI agents perform multi-turn refactoring tasks while potentially attempting to inject hidden vulnerabilities or "backdoors." The platform is configurable via YAML files and addresses limitations of existing research by providing a larger attack surface and supporting agent harnesses that use tools and persist state across interactions.

Refactor Arena specifically tests the "product threat vector"—where vulnerabilities are inserted into code that could be exploited later after deployment. This contrasts with "process attacks" that happen during execution. The setting acknowledges that modern coding agents from companies like OpenAI and Anthropic operate on larger codebases with hundreds of tool calls, requiring different monitoring protocols than single-pass models. Initial evaluations validate the platform, and researchers are encouraged to extend it for exploring control protocol overhead, adaptive attacks, and factored cognition in AI-assisted development.