Step-level Optimization for Efficient Computer-use Agents

Current computer-use agents invoke large multimodal models at nearly every step, making them expensive and slow for real-world software automation. Researchers have identified that long-horizon GUI tasks are highly heterogeneous—most steps are routine and can be handled by smaller, cheaper policies, while errors concentrate at a small number of high-risk moments. Two failure modes dominate: progress stalls (looping or ineffective actions) and silent semantic drift (continuing plausible actions after deviating from the user's true goal).

To address this inefficiency, the team proposes a step-level cascade that runs a lightweight policy by default and escalates to a stronger model only when needed. Two complementary learned monitors trigger escalation: a Stuck Monitor detects degraded progress from recent reasoning-action history, and a Milestone Monitor identifies semantically meaningful checkpoints where sparse verification is most informative. The framework is modular and deployment-oriented—it layers on top of existing computer-use agents without retraining the large model, turning always-on frontier-model inference into adaptive, on-demand compute allocation.