Spatial-Omni gives LLMs spatial hearing with 16 audio understanding subtasks
Researchers built a lightweight encoder that lets LLMs locate sounds and reason about spatial audio scenes.
Current multimodal LLMs process audio as monaural signals, discarding spatial cues for sound localization, spatial relation reasoning, and scene understanding. Spatial-Omni solves this with a lightweight SO-Encoder that injects First-Order Ambisonics (FOA) spatial audio into existing Omni LLMs as an independent modality—no modification to the original audio encoder needed. The encoder adds spatial tokens with minimal extra context and uses efficient staged training to improve spatial understanding.
To train and evaluate, the team built SO-Dataset from open-source data, real recordings, and simulations (400K FOA clips, 2.1M QA pairs), plus SO-Bench covering 16 spatial audio subtasks—from basic detection to complex reasoning. Experiments show Spatial-Omni beats existing large audio-language models (LALMs) and Omni LLMs on spatial tasks while maintaining general audio performance. Code and data are publicly available, opening the door for LLMs to perceive 3D sound environments in robotics, AR/VR, and accessibility.
- SO-Encoder integrates spatial audio as an independent modality without modifying existing audio encoders, using only limited additional context.
- SO-Bench benchmark covers 16 spatial audio understanding subtasks, including detection, location estimation, spatial relations, and complex reasoning.
- Spatial-Omni achieves state-of-the-art on spatial audio tasks while retaining competitive general audio understanding.
Why It Matters
Spatial hearing lets LLMs perceive 3D sound scenes, critical for robotics, AR/VR, and accessibility applications.