Microsoft's LoopsBench pushes coding agents to their limits
New benchmark reveals coding agents fail 75% of real-world dev tasks
Microsoft Research has unveiled LoopsBench, a groundbreaking benchmark that shifts coding agent evaluation from simple task completion to sustained long-horizon software development. Unlike traditional benchmarks focusing on isolated tasks, LoopsBench models real-world development as dependency graphs (DAGs) where agents must handle evolving requirements and regression testing.
The benchmark comprises 112 authentic tasks spanning 8 programming languages and 9 domains, with over 5,300 development units and executable tests. In evaluations of frontier coding agents, Anthropic's Opus-4.7 paired with Claude Code and outer continuation achieved the highest performance at just 25% task completion. Researchers found agent plans often missed critical dependencies, and regression issues persisted across loop implementations despite task completion.
- LoopsBench evaluates coding agents on long-horizon tasks with 112 real-world problems across 8 languages
- Only Anthropic's Opus-4.7 achieved 25% task completion rate in benchmark tests
- Benchmark includes dependency DAGs with regression testing and source-evident prerequisites
Why It Matters
This benchmark exposes critical gaps in AI's ability to handle real software development workflows, pushing the industry toward more robust coding agents.