ByteDance's SwanTale unifies text-driven and voice-cloned audio generation
One model handles voice design, cloning, and sound effects from natural language or reference audio
ByteDance's new SwanTale model tackles a growing challenge in audio production: the need to generate multi-speaker speech and sound effects for animation dubbing, audio dramas, movies, games, podcasts, and short videos. Creators often want to design a voice from scratch without a reference recording, control speaker style through natural language, and add acoustic scenes with environmental effects. SwanTale supports both instruct tasks, where a detailed caption drives generation, and zero-shot tasks, where a reference audio clip plus fine-grained text controls the output.
To make this work, the team built SwanData-Caption, a pipeline that cleans raw data, adds targeted synthetic coverage, and annotates multi-level captions. The model itself combines SwanVAE for high-quality multi-modality audio, Unified MoE for multi-task modeling, Engram conditioning, and reward-conditioned quality control. Training uses curriculum learning and GRPO post-training to progressively strengthen capabilities. SwanTale leads on multiple zero-shot and instruct metrics, achieves the best expressiveness scores in both tasks, and handles complex instruct generation involving multiple speakers and audio effects. Demos are publicly available.
- SwanTale unifies instruct and zero-shot multi-speaker audio generation in a single model, covering speech, sound effects, and acoustic scenes
- Introduces SwanData-Caption, an automated pipeline for cleaning, augmenting, and annotating diverse audio data with multi-level captions
- Uses SwanVAE, Unified MoE, Engram conditioning, curriculum learning, and GRPO post-training to achieve state-of-the-art expressiveness on both tasks
Why It Matters
One model can replace separate TTS, voice cloning, and sound design tools, dramatically simplifying audio production for creators.