OdinEval benchmark pushes LLM code repair in Odin language
New OdinEval benchmark tests LLM-based code repair with 168 real-world Odin bugs
A team of 11 researchers from multiple institutions (including ShanghaiTech University and Fudan University) has released OdinEval, the first reproducible benchmark for evaluating LLMs on repairing defects in the Odin programming language. Odin is a systems language gaining traction in game development and high-performance applications, but lacked specialized evaluation tools for AI-assisted debugging. The benchmark compiles 168 real-world defects from public Odin repositories, each including base and fixed commits, gold-standard patches, regression tests, and execution records.
The evaluation protocol requires tests to fail on the buggy code and pass after applying the gold patch. When no developer-written test exists, the team employs a black-box testing approach where three model instances review and revise tests based on execution feedback. In testing six leading models, Kimi-K3 achieved the highest repair success rate at 66.7% (Resolved score), while Qwen3.8-Max led with 96.4% reproducibility (Repro score). The benchmark includes frozen data, containerized toolchains, validators, and audit manifests to ensure reproducible research.
- OdinEval is the first benchmark specifically designed for LLM-based program repair in the Odin language, featuring 168 real-world bug instances with gold patches and regression tests.
- Kimi-K3 leads repair success at 66.7% Resolved score, while Qwen3.8-Max achieves 96.4% reproducibility in black-box testing scenarios.
- The benchmark provides containerized toolchains, frozen data, and audit manifests to ensure reproducible research across different environments.
Why It Matters
Bridges a critical gap in AI-assisted debugging for systems languages, enabling fair benchmarking of code repair capabilities beyond mainstream languages.