ADK Arena test: No single agent framework dominates — best beats 80% tasks, median hits 32%
A new paper benchmarks 51 agent frameworks and finds 5.6x cost differences and no clear winner.
Researchers introduced ADK Arena, a pipeline using LLM-as-a-Developer to evaluate 51 Python Agent Development Kits across 4 benchmarks (SWE-bench, τ²-bench, Terminal-Bench, MCP-Atlas). They found 57% generation success, 5.6x cost variation ($0.6-$3.4 per agent), and no single framework dominates: the best resolves 80% of tasks, but median framework only 32%. Documentation, source code, and parametric knowledge are largely substitutable, with genuine framework usage in a 28-40% band. The method provides a controlled measure of framework effectiveness.
- ADK Arena tested 51 Python agent frameworks across 4 benchmarks; only 57% of builds succeeded.
- Costs varied 5.6x ($0.60 to $3.40 per agent) but did not predict success; top framework hit 80% task resolution vs median 32%.
- Framework usage stayed in a narrow 28-40% band regardless of documentation or source code access — suggesting substitutability.
Why It Matters
With 51 ADKs on the market, this benchmark gives developers data-driven insight to pick the best framework for their agent tasks.