New paper rethinks continual learning for speech and audio models

A new paper from researchers Yang Xiao, Siyi Wang, Eun-Jung Holden, and Ting Dang tackles a critical blind spot in continual learning (CL) for speech and audio. Published on arXiv (2605.24863), the work argues that existing CL methods were designed for static task boundaries and isolated knowledge retention—assumptions that break down with modern speech foundation models. These models produce highly entangled, continuous representations that jointly encode linguistic content, speaker identity, and paralinguistic cues (e.g., emotion, prosody) within a shared latent space. The authors propose a representation-centric taxonomy that organizes CL approaches by how underlying representation geometry evolves under non-stationary acoustic conditions, rather than by task or data distribution shifts.

The paper highlights key mismatches: current CL techniques often assume task boundaries are known and that representations are separable, but speech models exhibit strong coupling between factors. For example, fine-tuning on a new speaker can degrade emotion recognition performance because both are encoded in overlapping subspaces. The authors outline open problems, including how to preserve shared structure without catastrophic forgetting, how to design CL algorithms that respect the geometry of acoustic manifolds, and how to evaluate CL in speech without artificial task splits. This work is a foundational step toward building audio systems that continuously adapt in real-world, non-stationary environments, such as voice assistants that learn new users without retraining from scratch.