ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systems

Researchers Akshey Sigdel and Rista Baral have introduced ToolMisuseBench, a new benchmark designed to rigorously evaluate how AI agents misuse tools and attempt recovery. The offline, deterministic benchmark covers four key environments—CRUD operations, retrieval, file systems, and scheduling—and includes a public dataset of 6,800 tasks. It injects replayable faults to simulate common operational failures like invalid arguments and interface drift, then measures performance against explicit budgets for steps, calls, and retries. The system reports on success rates, invalid call behavior, policy violations, recovery quality, and budget efficiency, providing a comprehensive framework for assessing agent robustness beyond simple language understanding.

Baseline results from the released benchmark show that while schema-aware methods achieve some fault-specific recovery gains, overall success remains limited under the tested authorization and hard failure settings. The reproducible evaluation pipeline and public dataset aim to standardize testing for tool-using agents, a critical need as AI systems move from chat interfaces to taking actions in real-world applications. This addresses a key gap in agent evaluation, where failures often occur for operational reasons even when the underlying language model performs well.