Stanford's OpaqueToolsBench exposes AI's struggle with real-world, poorly-documented tools

Researchers from Stanford and Allen Institute for AI created OpaqueToolsBench, a benchmark testing LLM agents on three opaque, real-world tool environments like search APIs and chess. Their results show current auto-documentation methods are expensive and unreliable. They propose ToolObserver, a framework that iteratively refines documentation by observing execution feedback, which outperforms baselines and uses 3.5-7.5x fewer tokens for exploration.