SALMONN-2 achieves SOTA hearing AI with self-supervised encoders
New ALLM matches or beats supervised models on speech, audio, music tasks
Researchers from multiple institutions introduce SALMONN-2, an audio large language model (ALLM) that leverages self-supervised learning (SSL) encoders instead of the typical supervised audio encoders. The key innovation is a multi-layer feature fusion (MLF) adapter that aggregates hierarchical representations from all encoder layers before projecting them into the language model. This approach allows SALMONN-2 to achieve performance comparable to or better than specialized supervised encoders across speech, audio, music, and paralinguistic tasks.
The model sets new state-of-the-art results on the MMAU-Pro, MMAR, and MMSU benchmarks among comparable-scale open-weight models. Additionally, the paper explores multimodal in-context learning (MICL) in ALLMs, finding that this capability does not emerge naturally but can be effectively acquired through targeted contextual biasing training. This demonstrates that SSL-based representations can serve as a powerful, general-purpose foundation for next-generation hearing AI without the need for massive supervised datasets.
- SSL encoder achieves comparable or better performance than supervised audio encoders across speech, audio, music, and paralinguistic tasks.
- Multi-layer feature fusion (MLF) adapter aggregates representations from all encoder layers before projecting into the language model.
- Multimodal in-context learning (MICL) is not emergent in ALLMs but can be acquired through contextual biasing training.
Why It Matters
SSL encoders could replace supervised ones, enabling more general-purpose, balanced audio AI without large labeled datasets.