New Test Shows AI Phone Helpers Lose Track While Using Apps

Companies are racing to build AI agents that can use your phone for you—book a table, order groceries, or fill out forms. To train and test those agents, researchers often rely on “world models”: AI systems that generate what the screen should look like after an action. But a new paper from academic researchers finds a serious problem. A world model may generate one beautiful, realistic screen and still be completely unreliable after a few steps. Think of an actor who delivers one convincing line but forgets the plot by the third.

The researchers introduce a test called GUI-CC. It checks whether an AI-simulated app stays consistent over an entire task, not just one screen. In one part, they rolled AI models along 500 real mobile-app paths from an existing dataset, GUIOdyssey. In another part, a separate probing AI interacted with generated screens in 200 emulator-verified tasks across 30 mobile apps. GUI-CC measures whether each transition looks realistic, whether the new screen plausibly follows from the last one, whether the task context is carried along, and whether the agent actually makes progress.

The key finding: producing one plausible screen does not mean the AI can reliably simulate a whole task. Current models often create screens that look usable, but they lose details from earlier steps—choices already made, buttons already pressed, or the original goal itself. That matters because these simulations are used to train and evaluate agents without needing a real phone. If the simulation drifts, an agent could look smart during testing and fail in the real world.

For everyday users, this is a reality check on the promise of “do it for me” phone assistants. Whether booking flights, filling forms, or troubleshooting an app, keeping context across many screens is essential. This benchmark pushes developers to test AI assistants the way they will actually be used—multi-step, messy, and easy to get lost.