CEAEval: New AI framework judges if speech fits the context
CEAEval-M uses reinforcement learning and multi-model collaboration to outperform existing systems.
Evaluating expressive speech has focused on emotional intensity, ignoring whether the delivery fits the surrounding narrative or interactive context. This gap limits the quality of audiobooks, conversational agents, and other speech-driven applications. To address this, a team of researchers from multiple institutions introduces CEAEval (Context-rich framework for Evaluating Expressive Appropriateness), which assesses if a speech sample aligns with the communicative intent implied by its discourse-level context.
To support the task, the team created CEAEval-D, the first context-rich speech dataset featuring real human performances in Mandarin conversational speech, with narrative descriptions and 15 dimensions of human annotations covering expressive attributes and appropriateness. They also developed CEAEval-M, a model combining knowledge distillation, planner-based multi-model collaboration, adaptive audio attention bias, and reinforcement learning. On a human-annotated test set, CEAEval-M substantially outperformed existing speech evaluation and analysis systems, marking a significant step toward more nuanced speech assessment.
- CEAEval evaluates speech expressive appropriateness considering discourse-level narrative context, not just emotional intensity.
- CEAEval-D is the first Mandarin conversational speech dataset with 15 annotation dimensions including expressive attributes and appropriateness.
- CEAEval-M integrates knowledge distillation, multi-model collaboration, adaptive audio attention bias, and reinforcement learning to achieve substantial performance gains over existing systems.
Why It Matters
Context-aware speech evaluation enables more natural audiobooks and conversational agents, improving user experience in narrative-driven applications.