BAT gives LLMs spatial hearing to reason about sound environments

Spatial sound reasoning—the ability to interpret where sounds come from and how they interact—is something humans do effortlessly, but AI has struggled with it. Addressing this, researchers from UT Austin developed BAT (Binaural Acoustic Transformer), a system that fuses a novel spatial audio encoder called Spatial-AST with the LLaMA-2 7B large language model. This lets the model not just detect sounds, but reason about their spatial relationships, such as inferring a conversation happening in a corner or a vehicle approaching from the left.

Because no existing dataset offered in-the-wild spatial sounds, the team synthesized binaural audio from AudioSet and SoundSpaces 2.0, then built SpatialSoundQA, a question-answering benchmark covering sound event detection, localization, distance estimation, and multi-sound reasoning. The Spatial-AST encoder alone achieves strong results on traditional Sound Event Localization and Detection tasks, but when paired with the LLM, BAT goes beyond simple detection. The paper reports superior performance across both perception and reasoning tasks, demonstrating that LLMs can meaningfully interpret complex spatial audio environments. BAT is accepted to ICML 2024, with code, datasets, and model weights publicly available.