CodecSep separates any sound with text prompts, using 54x less compute than AudioSep

A new paper from researchers Adhiraj Banerjee and Vipul Arora presents CodecSep, a prompt-driven universal sound separation framework that works directly in neural audio codec latent space. Unlike traditional systems like AudioSep that require decoding audio first, CodecSep combines a frozen DAC (descript audio codec) backbone with a lightweight FiLM-conditioned Transformer masker driven by CLAP text embeddings. This design allows open-vocabulary separation—users can describe any sound with text and extract it—while preserving the efficiency of a codec-native pipeline. The model separates sources through explicit latent masking rather than decoder-style generation, which the authors show is substantially more effective in codec space.

On benchmarks including dnr-v2 and five open-domain datasets, CodecSep consistently improves over AudioSep in SI-SDR (signal-to-distortion ratio) and achieves competitive ViSQOL scores with clear gains in human MOS-LQS. But the standout metric is compute: CodecSep requires only 1.35 GMACs end-to-end, roughly 54x less than AudioSep in the same pipeline and 25x lower for the separator alone, with significantly lower latency and memory. It also provides a practical deployment path for code-stream audio—when audio arrives as neural codec codes, CodecSep maps them to embeddings, separates in codec space, and outputs waveforms or re-quantized codes, avoiding the decode-separate-re-encode loop. This makes CodecSep a blueprint for efficient, codec-native downstream audio processing.