AppWorld-UL benchmark stumps Claude Opus 4.7 with 48.6% success rate

Current benchmarks for tool-use agents focus on executing commands without user interaction—but real-world tasks like ordering groceries often require back-and-forth clarification. To address this, Junzhi Chen and colleagues from multiple institutions built AppWorld-UL, a 'user-in-the-loop' benchmark comprising 516 challenging tasks. These tasks introduce ambiguities and constraints that force agents to ask the user questions, prompt for confirmation, or inform when a request is infeasible. The benchmark builds on the AppWorld environment, which includes nine popular simulated apps such as Amazon, Spotify, and Google Maps. User behavior is simulated via an LLM prompted with carefully designed knowledge boundaries, offering more reliable simulation than prior unconstrained or rigid approaches.

Results are sobering: Claude Opus 4.7, a state-of-the-art LLM, achieves only 48.6% success overall on AppWorld-UL. On the harder compositional subset—tasks requiring multiple steps and interactions—success drops to 35.7%. Even more telling, when evaluated on a stricter scenario-level metric (where both task completion and interaction quality are scored), compositional task performance plummets to just 21.3%. Analysis shows that correct user interaction is crucial for success: agents that fail to ask for clarification or that act without confirmation are penalized. The benchmark's difficulty and structured evaluation promise to drive advances in user-in-the-loop tool-use agents, a capability essential for practical AI assistants.