MIT's USAD 2.0 scales universal audio encoder to 1B parameters

Audio encoders are becoming critical as large language models (LLMs) increasingly rely on a single encoder for diverse inputs. While self-supervised learning (SSL) has produced strong domain-specific encoders (e.g., speech or music experts), multi-domain approaches like USAD and SPEAR have remained limited in coverage and evaluation. Recent studies also suggest that supervised encoders align better with audio LLMs. To address these gaps, MIT CSAIL researchers developed USAD 2.0, a universal audio encoder that distills knowledge from both SSL and supervised foundation models. The key innovation is domain-aware distillation, which handles teacher mismatch—a common issue when combining experts from different audio domains. USAD 2.0 also extends coverage to the music domain, which previous versions lacked, and introduces a second-stage supervised distillation step to improve downstream usability.

Scaling was achieved via depth scaling, pushing the model to one billion parameters. Experiments show USAD 2.0 achieves strong or state-of-the-art performance across a wide range of probing tasks and LLM-based evaluations. The paper was accepted to Interspeech 2026, signaling its impact on the audio AI community. By unifying speech, music, and general sound understanding into a single encoder, USAD 2.0 could simplify the architecture of voice assistants, music generation tools, and sound analysis pipelines—all while maintaining or improving accuracy.