VITA-QinYu: First AI model that role-plays and sings with human-like expression

Researchers have unveiled VITA-QinYu, the first end-to-end spoken language model (SLM) that goes beyond natural conversation to handle both role-playing and singing generation. Unlike previous models that treat speech as a single stream, VITA-QinYu uses a hybrid speech-text paradigm that interleaves text-audio modeling with multi-codebook audio tokens. This design enriches paralinguistic representation—capturing tone, mood, and performance nuances—while keeping modalities separate to avoid interference. To train this model, the team built a comprehensive data generation pipeline, synthesizing 15.8K hours of natural conversation, role-playing, and singing data.

On objective benchmarks, VITA-QinYu demonstrates superior expressiveness: it outperforms competing SLMs by 7 percentage points on role-playing tasks and scores 0.13 points higher on a 5-point Mean Opinion Score (MOS) for singing quality. Simultaneously, it sets new state-of-the-art results on conversational accuracy and fluency, beating prior SLMs by 1.38 percentage points on the C3 benchmark and 4.98 percentage points on URO. The model is fully open-source, with code, weights, and a demo supporting streaming and full-duplex interaction—making it a significant milestone in expressive AI speech.