Alibaba's Qwen-Audio-3.0 unifies speech, music, and sound effects in one model
A single Diffusion Transformer now generates complete 48kHz stereo audio scenes with multiple roles and domains.
Alibaba's Qwen team has introduced Qwen-Audio-3.0-Gen-Preview, a unified non-autoregressive framework for generating mixed audio across speech, music, sound effects, and ambience. The architecture employs a Diffusion Transformer (DiT) paired with a shared variational autoencoder (VAE) that compresses 48kHz stereo waveforms into compact 25Hz latent sequences. Prompt enhancement converts free-form user requests into structured temporal records, while a two-stage data curriculum and semantic conditional views train the model to apply these conditions across domains. This setup allows the system to generate long-form, temporally coherent audio scenes involving multiple roles and sound categories.
On benchmarks, the model shows clear strengths. It achieves top speaker similarity across all three subsets of Seed-TTS-Eval and demonstrates higher cross-turn consistency than Seed-Audio-1.0 in both English and Chinese multi-speaker scenarios. On AudioCaps, it excels in temporal localization, especially when evaluated with large audio-language models and AudioBox. Despite using only about 10% of the music data that a dedicated in-house model would require, Qwen-Audio-3.0 remains competitive across all seven SongBench components and even leads in three, all while preserving its speech and general-audio performance.
These results underscore the viability of unified, multi-domain audio generation for complex real-world use cases. By compressing stereo waveforms efficiently and leveraging a DiT-based generative backbone, the model simplifies the pipeline for creating rich audio content—from podcasts and virtual assistants to immersive soundscapes. The ability to handle mixed domains and temporal structure in a single framework could reduce the need for separate, domain-specific audio models, lowering both computational costs and development complexity.
- Uses a Diffusion Transformer (DiT) and shared VAE to generate mixed audio from speech, music, and sound effects in a unified framework.
- Compresses 48kHz stereo waveforms into 25Hz latent sequences, enabling efficient long-form scene generation.
- Outperforms Seed-Audio-1.0 in cross-turn consistency and temporal localization, and holds its own on music benchmarks with only 10% music training data.
Why It Matters
Unified audio generation simplifies creating complex soundscapes, reducing the need for separate models in content creation, virtual assistants, and immersive media.