Turing Test on Screen: A Benchmark for Mobile GUI Agent Humanization
New benchmark forces AI agents to mimic human touch dynamics to evade platform detection.
A research team from Shanghai Jiao Tong University and other institutions has published a paper titled 'Turing Test on Screen: A Benchmark for Mobile GUI Agent Humanization.' The core argument is that as platforms deploy adversarial countermeasures, autonomous agents must evolve beyond raw utility to include 'humanization' capabilities for survival. The researchers formally model this as a MinMax optimization problem between a detector and an agent aiming to minimize behavioral divergence.
To ground their work, the team collected a new high-fidelity dataset of human mobile touch dynamics. Their analysis revealed that current agents, particularly those based on Large Multimodal Models (LMMs), are easily flagged due to unnatural, robotic kinematics. In response, they established the Agent Humanization Benchmark (AHB) with specific detection metrics to quantify the trade-off between an agent's imitability and its task performance.
The paper concludes by proposing practical methods to bridge this gap, ranging from simple heuristic noise injection to sophisticated data-driven behavioral matching. The researchers demonstrate that agents can theoretically and empirically achieve high levels of human-like imitation without a significant drop in utility. This work represents a paradigm shift, moving the field's focus from whether an agent can complete a task to how seamlessly it can operate within human-centric digital ecosystems.
- Proposes the 'Turing Test on Screen' and Agent Humanization Benchmark (AHB) to measure how human-like an AI agent's screen interactions are.
- Finds vanilla LMM-based agents are easily detectable due to unnatural touchscreen kinematics, using a new dataset of human mobile touch dynamics.
- Demonstrates methods like behavioral matching can help agents evade detection without sacrificing task performance, enabling stealthier automation.
Why It Matters
Enables the development of AI agents that can operate undetected on platforms, crucial for automation, testing, and assistive tools in restrictive digital environments.