Prompt Amplification and Zero-Shot Late Fusion in Audio-Language Models for Speech Emotion Recognition

A new research paper introduces ZS-Fuse, a method that significantly improves AI's ability to recognize emotions in speech. The core challenge is that while general-purpose Audio-Language Models (ALMs) like OpenAI's Whisper are versatile, specialized Foundation Models (FMs) still outperform them on specific tasks like Speech Emotion Recognition (SER). ZS-Fuse bridges this gap through a 'late-fusion' approach, strategically combining the zero-shot capabilities of ALMs with the precision of specialist FMs to achieve new state-of-the-art results.

The method tackles two major hurdles in zero-shot SER: ambiguity in emotional labels and high sensitivity to the wording of text prompts. To solve this, the researchers employ a simple prompt ensemble and a novel technique called 'prompt amplification,' which involves repeating audio and text queries to uncover the model's strongest capabilities. When tested with three dual-encoder ALMs and two FMs, ZS-Fuse demonstrated consistent improvements over leading baselines, including the powerful WavLM-Large model, across three standard SER datasets.

This work is significant because it provides a clear, effective blueprint for leveraging the flexible, instruction-following nature of modern ALMs without sacrificing the accuracy required for critical real-world applications. It moves beyond treating models in isolation, showing how hybrid architectures can extract superior performance. The techniques of prompt amplification and late fusion are likely to influence how developers build more reliable and adaptable audio AI systems for healthcare, customer service, and content analysis.