Beyond Screenshots: Evaluating VLMs' Understanding of UI Animations

Researchers from the University of Michigan and other institutions have introduced AniMINT, a novel dataset of 300 densely annotated UI animation videos, to evaluate how well Vision Language Models (VLMs) understand dynamic user interfaces. The study, accepted at ACL 2026, systematically tests state-of-the-art VLMs on three levels: perceiving animation effects (e.g., motion type), identifying animation purposes (e.g., feedback or state change), and interpreting animation meaning (e.g., what the animation communicates to the user). Results show that while VLMs can reliably detect primitive motion—like sliding or fading—their high-level interpretation remains inconsistent and substantially below human performance.

The team also introduced Motion, Context, and Perceptual Cues (MCPC) to probe factors affecting VLM performance. Key bottlenecks include difficulty integrating temporal context and understanding the functional role of animations beyond surface-level changes. This work underscores a critical gap in current AI agents' ability to interact with modern interfaces, where animations serve essential communicative purposes, not just aesthetics. The findings point to a clear need for improved temporal reasoning and contextual grounding in future VLMs to enable truly reliable UI agents.