Improving quality and robustness in LLM-based text-to-speech systems

Amazon's AI research team has unveiled a comprehensive solution to three persistent problems plaguing modern LLM-based text-to-speech systems: accent leakage in multilingual voice cloning, robotic expressiveness, and unreliable generation that leads to hallucinations or premature cutoffs. Their approach uses low-rank adaptation (LoRA) to fine-tune polyglot models on locale-specific data, effectively separating speaker identity from accent. This allows a voice cloned from English audio to speak Spanish or German with native pronunciation while preserving the original speaker's vocal characteristics.

To tackle expressiveness, the team employs classifier-free guidance (CFG) to generate synthetic reference audio with enhanced emotional cues like laughs and sighs, teaching the model to adopt more natural prosody. For reliability—a critical weakness where autoregressive models can generate confident-sounding nonsense or stop mid-sentence—they implement chain-of-thought reasoning. Before generating speech tokens, the model first predicts phoneme sequences and estimates durations, creating an explicit plan that prevents hallucinations and truncations.

The results are measurable: in MUSHRA listening tests across nine locales including US-English, Germany-German, and Spain-Spanish, the new system showed quality improvements ranging from 5.5% to over 20% compared to previous models. This represents a significant step toward truly robust, expressive, and accent-free polyglot TTS that can scale a handful of recorded voices to dozens of languages while maintaining naturalness and reliability.