AGMem boosts GUI agent success by 33% over naive visual memory
Storing full screenshots actually makes GUI agents worse at taking actions.
Researchers from KAIST (Seoyoung Choi et al.) conducted a systematic failure-mode study of GUI agents that use visual memory—storing screenshots from past interactions to guide future decisions. They created a taxonomy of four failure types: cognitive failure (wrong plan), visual state misunderstanding (misreading the screen), hidden operation blindness (missing invisible UI elements), and grounding error (clicking wrong coordinates). Surprisingly, they discovered that prepending full-image memory has a divergent effect—it reduces state-level failures (visual misunderstanding) but worsens action-level failures (hidden operation blindness and grounding error). This reveals that naive visual memory provides too much irrelevant context, distracting the agent from precise actions.
To address this, the team proposed AGMem (Action-Grounded Visual Memory), which stores only cropped images of the GUI regions directly related to successful actions or recovery steps, rather than full screenshots. This focuses the agent on locally relevant visual cues while discarding noise. Experiments on the OSWorld benchmark showed AGMem improved task success rates by 33.3% over full-image memory, demonstrating that smarter memory representation—not just more data—is key to making GUI agents reliable. The paper is set for ICML 2026 Workshop.
- Four failure modes identified: cognitive failure, visual state misunderstanding, hidden operation blindness, grounding error
- Full-image memory cuts state-level errors by 12% but increases action-level errors by 18%
- AGMem uses action-grounded crops instead of full screenshots, boosting task success by 33.3% on OSWorld
Why It Matters
Smarter visual memory design could make GUI agents more reliable for automating real-world computer tasks.