MetaSpace testing framework exposes spatial blindspots in embodied AI agents

MetaSpace, a new framework from researchers including Gengyang Xu, Dongwei Xiao, Yiteng Peng, and Shuai Wang, tackles a growing problem in embodied AI: how to reliably test spatial reasoning. Current evaluation methods fall into two camps—manually annotated visual question answering pairs (labor-intensive and inconsistent) and high-level task completion metrics like navigation success (which can hide unsafe shortcuts or suboptimal behaviors). Spatial cognition is the foundation for any embodied task, yet it hasn't been directly and systematically assessed.

MetaSpace applies metamorphic testing, a software engineering method, to generate test cases from spatiotemporal multimodal states captured during real agent executions. The framework encodes metamorphic relations (MRs)—logical and physical laws—as executable Prolog rules. When an agent's behavior violates an MR, it signals a spatial cognition failure. Running across three embodied scenarios, MetaSpace detected 90,422 such errors in state-of-the-art MLLM-driven agents. The resulting Spatial Cognition (SC) score quantifies performance, and every SOTA agent averaged between 0.44 and 0.52—dramatically below the human benchmark of 0.96. These findings suggest current benchmarks may be masking severe spatial reasoning gaps in embodied AI, which could translate to safety risks in real-world robotics and physical interactions.