SemaVoice TTS slashes word error rate to 1.71% with semantic-aware alignment

A team of researchers from multiple Chinese institutions (including authors affiliated with a speech technology company and academic labs) has introduced SemaVoice, a novel zero-shot text-to-speech (TTS) framework that tackles the fundamental mismatch between semantic-prosodic modeling and reconstruction-driven continuous speech representations. Current continuous autoregressive TTS models tend to overfit low-level acoustic textures, sacrificing high-level semantic coherence and accumulating errors during generation. SemaVoice solves this with a Speech Foundation Model (SFM) guided alignment mechanism that refines continuous speech representations to preserve both local semantic consistency and global structural relationships. These refined representations then feed a patch-wise diffusion head within the autoregressive loop.

On the Seed-TTS benchmark, SemaVoice achieves an impressive 1.71% English word error rate (WER), making it highly competitive with state-of-the-art open-source systems in both objective and subjective evaluations. The paper further validates the SFM guided alignment's effectiveness by showing significant improvements under varying representation granularities while keeping an information-rate constraint. This breakthrough could accelerate progress in voice cloning, personalized digital assistants, and accessibility tools, offering more natural and semantically precise speech synthesis without requiring speaker-specific training data.