PhoneHarness benchmark boosts phone agents with mixed GUI, CLI, and tool actions

Current phone agents are largely evaluated as GUI controllers—observing screens and emitting taps and swipes. But real mobile workflows demand more: deciding when to use app GUIs, device-side commands, or structured tools, and leaving evidence that the intended effect occurred. To address this, Chenxin Li and 20 co-authors from multiple institutions introduced PhoneHarness, a mixed-action benchmark and execution harness. It runs a device-side agent loop over GUI, CLI, and host-side tool actions, combining deterministic action routing with bounded GUI delegation and auditable execution traces. On the annotated evaluation split, PhoneHarness reaches a 75.0% pass rate, outperforming the strongest non-PhoneHarness settings by 12.9 percentage points. The benchmark, PhoneHarness Bench, evaluates whether agents complete tasks with observable side effects, not just whether they produce plausible final answers.

PhoneHarness and PhoneHarness Bench play distinct but mutually dependent roles: the harness makes mixed phone workflows executable, while the benchmark measures whether agents can use that harness reliably and safely. The findings suggest that reliable phone automation depends on action-surface routing and verifiable execution, not just visual GUI control. This represents a significant step toward practical phone agents that can handle real-world tasks—from automating app interactions to executing device commands—with accountability and traceability. By focusing on verifiable outcomes, PhoneHarness sets a new standard for evaluating mobile agents, with implications for both research and commercial deployment of phone automation systems.