StateAct boosts Claude Opus 4.8 by using program state, not pixels

A new paper from researchers including Yan Yang, Silvio Savarese, and Junnan Li introduces StateAct, a multi-agent architecture that fundamentally rethinks how computer-use agents interact with software. Instead of relying on screenshots—a lossy rendering of underlying program state—StateAct's main agent works directly with program state via code, accessing files, application backends, and DOM. A dedicated GUI subagent handles screenshot-and-click interaction only for the rare subgoals that truly need it: just 28 of 108 tasks and 1.1% of main-agent steps.

StateAct also incorporates a verification gate that double-checks the final output for structural failures (missing, unsaved, or wrong-path files). On the OSWorld 2.0 benchmark, the system lifts Claude Opus 4.8 from 20.6% to 26.9% binary success and from 54.8% to 61.8% partial success—at roughly 9x lower cost than the same model driven solely by screenshots. A code-only variant without the GUI subagent achieved only 45.9% partial success, below the screenshot baseline. The authors argue that grounding action, verification, and memory in program state—what they call state-grounding—shifts the main bottleneck from perception toward reasoning: failures now depend more on what the agent thinks than on what it sees.