UIUC-IBM study: self-evolving coding agents share universal playbook across 8 languages

Self-evolving harnesses—closed-loop systems where an AI agent inspects its own outputs and edits its prompts, tools, and memory—have shown reliable gains on coding benchmarks. But until now, no one knew what those evolved artifacts actually encode: benchmark-specific hacks, language-specific engineering knowledge, or compensation for model weaknesses. Researchers Siqi Yang, Qianlan Yang, Yu-Xiong Wang, Saurabh Pujar, and Martin Hirzel (UIUC and IBM Research) set out to disentangle these factors by holding one evolution recipe fixed across a grid of eight programming languages from Multi-SWE-Bench and three base models.

Their approach routes every edit through a typed failure signal, recorded as a falsifiable contract, making each modification attributable. Four key findings emerged. First, the loop improves held-out solve rates over both minimal and manually designed scaffolds in most cells—but with two null regions. Second, gains directly compensate for recoverable execution defects: where defect mass is near zero, gain is near zero. Third, evolved harnesses share an abstract playbook across languages but instantiate it with almost entirely disjoint language ecosystem machinery. Finally, that shared core transfers and can be distilled into one universal harness, while the ecosystem margin resists distillation and requires native re-evolution. This reframes evolved harnesses as legible compensation layers shaped jointly by language engineering demands and model behavioral gaps—not opaque benchmark-tuned scaffolds.