Tool-calling benchmarks flawed: 18.5% misalignment found in audit

A new validity audit of tool-calling benchmarks by Bhat et al. (arXiv:2607.02577) finds that current evaluation methods may significantly overstate or distort model capabilities. After analyzing 496 expert-reviewed tasks across four major benchmark families—BFCL v4, τ²-Bench, LiveMCPBench, and MCP-Atlas—the team identified 92 evaluator-human disagreements, an 18.5% misalignment rate. The failures are not random: deterministic benchmarks suffer from brittle state matching, trajectory lock-in, and incorrect ground truths, while LLM-judge benchmarks exhibit rubric drift, hallucinated completions, and answer-only scoring.

Most striking is the run-to-run variance in LiveMCPBench, where 23 repeated evaluations of the same setup produced scores ranging from 57.9% to 76.8%—an 18.9 percentage point spread large enough to change leaderboard conclusions. To address these issues, the authors introduce Tool-Veritas, a configurable benchmark combining deterministic state verification with optional qualitative judging, and Harness Lab, an open-source system for trace inspection, repeated-run comparison, and evaluator debugging. They argue for decomposed metrics that separately measure tool invocation, task completion, and outcome verification, warning that current scores may reflect evaluator artifacts rather than genuine agent capability.