ProsoCodec preserves prosody in voice conversion

Neural speech codecs have become foundational for speech generation, but their holistic representations intertwine content, speaker identity, and prosody—hindering tasks like voice conversion that require prosody transfer. ProsoCodec addresses this by modeling prosody as a conditional residual rather than a disentangled stream. By conditioning both encoder and decoder on text and speaker embeddings as prefix tokens, the discrete bottleneck captures only the prosodic variation not explained by content and speaker identity. This design preserves natural rhythm, stress, and intonation when converting voices.

To further enhance prosody fidelity, ProsoCodec leverages low-frequency mel bands—which carry most prosodic information—and trains on paired same-speaker utterances (same speaker saying the same phrase with different prosody). Experiments show significant improvements in prosody preservation while reducing source-timbre leakage (the target voice unintentionally sounding like the source voice). The model achieves this without complex adversarial training or auxiliary networks. ProsoCodec is accepted at Interspeech 2026 and opens new possibilities for expressive voice conversion, dubbing, and assistive communication systems.