Schema First Tool APIs for LLM Agents: A Controlled Study of Tool Misuse, Recovery, and Budgeted Performance

Researchers Akshey Sigdel and Rista Baral conducted a controlled study examining how different API interface designs affect LLM agent performance. The paper, 'Schema First Tool APIs for LLM Agents: A Controlled Study of Tool Misuse, Recovery, and Budgeted Performance,' tested three interface conditions: free-form documentation, JSON Schema specifications, and JSON Schema with structured validation diagnostics. All conditions preserved identical tool semantics and information content, allowing researchers to isolate interface design as the experimental variable.

Using a deterministic software engineering sandbox environment with logs, metrics, and repository tasks, the researchers ran a fully crossed pilot study with one open local model, three random seeds, three interface conditions, and four interaction budgets. They measured end task success, interface misuse, execution failures, semantic misuse, recovery behavior, and computational overhead. The results showed that while schema-based conditions significantly reduced interface misuse, they had no impact on semantic misuse or overall task success rates, which remained at 0% across all conditions.

The study's key finding is that interface formalization through JSON Schema improves contract adherence but doesn't address the fundamental bottlenecks in semantic action quality and timeout-sensitive tasks. This suggests that simply improving API documentation won't solve the core challenges facing LLM agents in complex software engineering environments. The research provides empirical evidence that semantic understanding and reasoning capabilities remain the primary limitations under constrained local inference budgets.