Joint Residual Reweighting boosts speaker similarity in F5-TTS, CosyVoice2

Zero-shot text-to-speech (TTS) systems can clone a voice from just a few seconds of audio, but they often struggle to balance how accurately they reproduce the speaker’s voice with how correctly they pronounce the given text. Standard classifier-free guidance (CFG) strengthens both conditions together, while recent branch-selective guidance tries to emphasize text or speaker separately—but that usually sacrifices one for the other. A new paper from Runwu Shi and colleagues revisits the problem by decomposing the guidance field into text, speaker, and joint residuals. They show that conventional speaker-selective guidance actually entangles the speaker residual with the joint residual, which can distort text-related generation.

The proposed solution—joint residual reweighting—lets the model independently control the strength of the speaker residual and the joint residual within the standard CFG framework. This simple change avoids the disruptive coupling seen in earlier methods. Experiments on F5-TTS and CosyVoice2 demonstrate that the technique improves speaker similarity significantly while keeping text correctness at competitive levels. The work provides a practical way to get the best of both worlds in zero-shot voice cloning, making AI voices sound more like the target speaker without garbled words or mispronunciations.