JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments

A research team led by Zhan Liu has introduced JAEGER, a groundbreaking framework that enables AI systems to perceive and reason about physical environments in three dimensions. The system addresses a critical limitation in current audio-visual large language models (AV-LLMs), which are confined to 2D perception using standard RGB video and monaural audio. This dimensionality mismatch has prevented reliable source localization and spatial reasoning in complex environments.

JAEGER's technical innovation centers on integrating RGB-D (depth-aware) visual observations with multi-channel first-order ambisonic audio, creating a unified 3D representation. The team developed Neural Intensity Vector (Neural IV), a learned spatial audio representation that encodes robust directional cues to enhance direction-of-arrival estimation, even in challenging acoustic scenarios with overlapping sound sources. To train and evaluate the system, they created SpatialSceneQA, a comprehensive benchmark containing 61,000 instruction-tuning samples curated from simulated physical environments.

Extensive experiments demonstrate that JAEGER consistently outperforms 2D-centric baselines across diverse spatial perception and reasoning tasks. The framework represents a significant step toward AI systems that can truly understand and interact with physical spaces, with implications for robotics, augmented reality, and autonomous systems. The researchers plan to release source code, pre-trained model checkpoints, and datasets upon acceptance, potentially accelerating development in embodied AI and spatial computing applications.