Discrete Token Modeling for Multi-Stem Music Source Separation with Language Models

A research team has introduced a novel AI framework that fundamentally rethinks how to separate the individual components of a mixed music track. Instead of directly predicting continuous audio signals, their method, detailed in the paper "Discrete Token Modeling for Multi-Stem Music Source Separation with Language Models," treats the task like generating text. It first encodes the mixed audio into a sequence of discrete tokens using a specialized neural audio codec called HCodec. A decoder-only language model then generates tokens for four target stems—like vocals, drums, bass, and other instruments—conditioned on the input mix, in an autoregressive fashion similar to how GPT models predict the next word.

This generative approach, evaluated on the standard MUSDB18-HQ benchmark, achieves perceptual quality that approaches state-of-the-art discriminative models. Crucially, it attained the highest NISQA (a perceptual speech quality metric) score on the separated vocals track, indicating superior sound quality for the most critical element in many mixes. The researchers also validated the importance of their learnable Conformer-based encoder and the benefit of generating tracks sequentially, allowing the model to leverage information from previously separated stems. This work represents a significant shift, applying the powerful sequence modeling capabilities of large language models to the complex domain of high-fidelity audio generation and manipulation.