DDBench benchmark shows LLM agents lag on distributed bugs
60 real bugs from 13 systems, 61-point pass rate gap across models.
Researchers introduce DDBench, a code-repair benchmark built from 60 historical bugs mined from 13 open-source distributed systems and split into three difficulty tiers. Testing 10 LLMs under symptom-only versus context-augmented conditions, they found pass rates spanned 61 percentage points, and adding bounded debugging context lifted aggregate pass rates by +18.1 pp. The gains were asymmetric: weaker models improved in pass rate, while stronger models gained efficiency, and even faithful debugging context sometimes misled the agents.
- DDBench contains 60 historical bugs from 13 open-source distributed systems, split into three difficulty tiers
- Pass rates among 10 LLMs spanned 61 percentage points, with 9 of 15 top model pairs significantly separated on hardest cases
- Bounded debugging context lifted aggregate pass rates by +18.1 pp, but gains were asymmetric and context occasionally misled agents
Why It Matters
As AI agents move from single-process coding to real-world distributed systems, DDBench exposes blind spots and shows context quality is decisive.