CUADebug: AI agents get smarter debugging tool

Researchers from the University of Illinois Urbana-Champaign and collaborators developed **CUADebug**, a framework designed to diagnose and repair failures in computer-use agents (CUAs)—AI systems that interact with desktop and web interfaces via screenshots, mouse/keyboard actions, and UI feedback. Unlike traditional text-only agents, CUAs fail due to complex, multimodal issues spanning visual perception, spatial grounding, low-level interactions, task reasoning, and dynamic environments. CUADebug tackles this by introducing a CUA-specific error taxonomy, **CUAErrorBench**, a human-annotated benchmark of 204 failed trajectories in the OSWorld environment, and **CUADebugger**, a tool-augmented debugger.

**CUADebugger** actively inspects suspicious steps by analyzing paired before/after screenshots and action traces, then generates structured diagnoses identifying root-cause steps, error types, grounded evidence, and corrective strategies for re-execution. In evaluations, it improved joint subtype-and-step diagnosis accuracy from 11.2% to 19.6% when paired with Gemini 2.5 Pro. Notably, it achieved a 25.86% success rate in continual re-execution tasks—more than double the baseline—while human-oracle guidance reached 29.21%. The framework’s root-cause analysis (RCA) conditions also outperformed history-only continuation methods (29.90% vs. 13.89% task completion).