MiDashengLM-Gen generates any audio scene from text
One model to rule them all: speech, music, sound effects in one pipeline
A team spanning 10 institutions—including Sun Yat-sen University and ByteDance AI Lab—has released MiDashengLM-Gen, an end-to-end framework that couples a pre-trained LLM with conditional flow matching to generate mixed-audio scenes from raw text prompts. Unlike prior approaches that chain frozen text encoders with separate audio decoders, MiDashengLM-Gen trains a single model to autoregressively produce variable-length speech, music and sound effects in one coherent pipeline. The method leverages per-token conditional flow matching to better align text semantics with audio tokens, yielding a 77% reduction in English Word Error Rate (WER) on the Seed-TTS benchmark—dropping from 12.15% to 2.79% and approaching the 1.24% WER of dedicated Text-to-Speech systems.
The framework generalizes beyond English: in multilingual tests it achieves competitive WERs versus specialized baselines, while maintaining high-quality mixed-audio generation on the MECAT benchmark. All code, model checkpoints and interactive demos are open-sourced on Hugging Face Spaces and Replicate, lowering the barrier for researchers and developers to build next-generation audio-generation applications.
- MiDashengLM-Gen is the first end-to-end LLM for unified text-to-audio generation, replacing disjoint pipelines with one trainable model.
- English WER improves from 12.15% to 2.79% (77% reduction) on Seed-TTS, nearing dedicated TTS performance (1.24%).
- Open-sourced code, checkpoints and demos on Hugging Face and Replicate for immediate experimentation.
Why It Matters
One model to generate speech, music and sound effects unlocks faster, cheaper production of immersive audio for games, film and AR/VR.