ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining

A team of researchers including Anuj Diwan, Eunsol Choi, and David Harwath has unveiled ParaSpeechCLAP, a novel AI architecture designed to bridge the gap between raw audio and rich linguistic descriptions of style. Unlike standard speech models focused on transcription, this dual-encoder contrastive model learns a shared embedding space for speech and text captions that describe a wide array of stylistic attributes. These include intrinsic features like a speaker's timbre and situational features like the emotion, pitch, or texture of a specific utterance. The team trained specialized models for each category (ParaSpeechCLAP-Intrinsic and ParaSpeechCLAP-Situational) as well as a unified ParaSpeechCLAP-Combined model, finding that specialization boosts performance on individual dimensions while the combined model handles compositional tasks best.

ParaSpeechCLAP demonstrates superior performance across three key applications: retrieving text captions that match a speech clip's style, classifying speech attributes, and serving as an inference-time reward model. This last function is particularly impactful for generative AI, as it allows the model to guide and improve the output of style-prompted text-to-speech (TTS) systems without requiring those systems to be retrained. By releasing the models and code publicly, the researchers provide a powerful new tool for creating more expressive, controllable, and human-like synthetic speech, moving beyond robotic monotones to captures nuances like sarcasm, urgency, or warmth.