Adversarial Critic Loop Kills 105 Mutants Missed by One-Shot AI Tests

Jeff Otterson from Arizona State University’s W. P. Carey School of Business has released a pre-registered study on adversarial test-hardening for AI-written code. The method uses a three-part loop: a Tester LLM writes an initial test suite, mutation testing identifies surviving injected defects, and a Critic LLM writes new tests specifically to kill those survivors—all verdicts are decided mechanically, so no model judges another’s output. On five Python subjects, the loop killed 105 mutants that one-shot generation missed and lost none. A pre-registered frozen-shared-round-0 design across four subjects found a mean incremental kill rate of 78.3% (95% CI: [0.592, 0.935]). A cross-provider configuration showed a positive pilot difference of 0.178 (95% CI: [0.039, 0.347]), but the magnitude was dominated by a single replicate and came at 5.5x lower arm cost.

The paper’s central contribution is an “instrument autopsy”: an earlier analysis reporting a cross-lineage effect at p = 9.5e-66 turned out to be an artifact of an output cap silently truncating the verbose model. The artifact was caught only through adversarial review of the completed analysis, and further review revealed that each arm had been resampling its own initial suite—a confound removed in Experiment 2. Otterson warns that cross-model comparisons can inherit asymmetries from the harness that runs them, including truncation recurrences and operational failures. The paper releases all protocols, run receipts, and analysis code to enable replication. The work highlights both the promise of automated, adversarial test hardening and the critical need for rigorous evaluation design when benchmarking AI systems.