LAIP unlocks spatial grounding from audio-visual retrieval models

Weakly supervised audio-visual sound source localization (SSL) has long been constrained by the need for dense spatial annotations. While large-scale retrieval models encode rich multimodal structure, their latent representations lose spatial detail in upper layers due to global pooling. Researchers from Telecom Paris and LTCI introduce LAIP (Localization via Audio-Informed Pooling), a framework that recovers this lost information by replacing the standard global aggregation module with a lightweight Audio-informed Spatial Pooling (AiSP) module. AiSP uses frame-aligned audio features to query intermediate visual tokens from the frozen retrieval backbone, preserving localized spatial structure that would otherwise be discarded.

LAIP achieves state-of-the-art performance on the AVSBench dataset and nearly doubles previous results on the challenging AVATAR benchmark. The method is computationally efficient — the AiSP module adds only a few hundred parameters — and requires no retraining of the backbone. This demonstrates that accurate spatial grounding does not require learning from scratch; it can be extracted from pre-trained retrieval representations, unifying retrieval and localization tasks. The paper (arXiv:2607.24786) suggests a path toward scalable, annotation-light SSL systems.