Selective Classifier-free Guidance for Zero-shot Text-to-speech

A new research paper from authors John Zheng and Farhad Maleki tackles a core challenge in zero-shot text-to-speech (TTS): balancing fidelity to a target speaker's voice with accurate adherence to the written text. The team investigated applying classifier-free guidance (CFG), a technique successful in AI image generation, to the speech synthesis domain. Surprisingly, they found that CFG strategies effective for images generally fail to improve speech quality, highlighting the unique complexities of audio generation.

Their solution is a novel 'selective' CFG approach. The method applies standard CFG during the early timesteps of the audio generation process and then strategically switches to a selective guidance mechanism only in the later stages. This hybrid technique improved speaker similarity scores by approximately 20% while successfully limiting the degradation of text adherence that usually comes with such adjustments. A key, unexpected finding was that the strategy's success is highly dependent on the text representation, producing different optimization results for English and Mandarin even when using the same underlying AI model.