Generalizable Audio-Visual Navigation via Binaural Difference Attention and Action Transition Prediction

Researchers Jia Li and Yinfeng Yu have introduced a novel framework called Binaural Difference Attention with Action Transition Prediction (BDATP) to solve a critical weakness in Audio-Visual Navigation (AVN). In AVN, AI agents must locate sound sources, like a ringing phone, in complex 3D spaces using only visual and auditory inputs. Current models often fail in new environments because they over-rely on recognizing specific sound types (semantic features) and memorize training layouts. The BDATP framework tackles this with a two-pronged approach: its Binaural Difference Attention (BDA) module mimics human hearing by focusing on the subtle timing and volume differences between a simulated agent's 'ears' to pinpoint a sound's direction, reducing dependency on what the sound is. Simultaneously, an auxiliary Action Transition Prediction (ATP) task forces the agent to predict its next move, acting as a regularizer to prevent it from learning shortcuts specific to its training environment.

Extensive testing on standard 3D simulation datasets like Replica and Matterport3D shows BDATP's superior generalization. The framework can be integrated into various existing navigation AI backbones, consistently boosting their performance. Its most impressive result is on the Replica dataset with 'unheard sounds'—audio categories not present during training—where it achieved a remarkable 21.6 percentage point absolute improvement in Success Rate. This demonstrates robust spatial reasoning divorced from semantic knowledge. The paper has been accepted for publication at the International Joint Conference on Neural Networks (IJCNN 2026), signaling peer-reviewed validation of its significance for embodied AI research.