AI That Clicks for You Stumbles on Pop-Ups and Clutter

Researchers built RealGUINoise, an interactive, cross-platform, extensible benchmark for testing how GUI agents and Computer-Using Agents hold up under the messy conditions of everyday interfaces. Existing benchmarks largely assume clean environments or focus narrowly on security, with no unified framework for consistent, automated end-to-end evaluation across agents and platforms. RealGUINoise spans 42 noise types across web, desktop and mobile tasks, integrates 7 representative agent frameworks, and evaluates them on real-world daily tasks through real-time interaction, comparing results against task-specific golden rubrics and clean-environment trajectories on reliability, safety and trajectory-level behavior. The experiments found the noise not only hurts task performance but substantially redirects agents' action trajectories and increases unsafe behavior, exposing a gap between clean-environment capability and dependable real-world operation.