GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents

A research team from the National University of Singapore and other institutions has introduced GameWorld, a new benchmark designed to standardize the evaluation of multimodal AI agents in video game environments. The benchmark addresses current limitations in testing these agents, which include heterogeneous action interfaces and heuristic verification methods. GameWorld features 34 diverse games spanning multiple genres, with 170 specific tasks, each paired with state-verifiable metrics for outcome-based evaluation. This allows for reproducible and rigorous testing of agents' abilities in fine-grained perception, long-horizon planning, and precise control.

The study evaluates two distinct agent interfaces: computer-use agents that directly emit keyboard and mouse controls, and generalist multimodal agents that act through a semantic action space using deterministic Semantic Action Parsing. Testing 18 different model-interface combinations revealed that even the top-performing agents are far from achieving human-level capabilities in video games. The researchers conducted extensive experiments with repeated full-benchmark reruns to demonstrate the robustness of their evaluation framework, while additional studies on real-time interaction, context-memory sensitivity, and action validity exposed further challenges for developing capable game-playing AI.