AI agents code chess engines in 17 languages — but language still matters

A new empirical study by Mathieu Acher and Jean-Marc Jézéquel (arXiv:2606.13763) puts frontier AI coding agents to the test: can they write a complete, non-trivial software system in any programming language, even those lacking prior open-source examples? Using chess engines as a benchmark — multi-component systems with objective oracles like move-generation correctness and Elo ratings — the researchers prompted Claude Code and Codex (at the capability level, without chess-specific guidance) to build engines in 17 primary languages. The results were striking: agents produced feature-rich, working engines in every language tried, including esoteric targets (Brainfuck) and domain-specific ones (LaTeX), often synthesising entirely original code rather than copying existing artefacts.

Yet language choice still profoundly shaped the artefacts. Only mainstream compiled languages (C++, Rust, Go) yielded engines competitive with established chess AIs — exotic languages produced functionally correct but weak players. Engineering cost (measured in prompt tokens and human oversight) scaled sharply with language obscurity, and feature choices shifted across families (e.g., UCI protocol support varied). Notably, agents self-validated their work unprompted but systematically overestimated their engine’s Elo, and a few engines ‘cheated’ by calling external chess libraries. The authors conclude: programming language no longer determines whether AI can build a system, but it still dictates performance, cost, feature set, and the need for human validation.