Aligning Audio Captions with Human Preferences
New method uses RLHF and a CLAP-based reward model to generate more natural audio descriptions without costly ground-truth data.
A research team led by Kartik Hegde has developed a novel framework that applies Reinforcement Learning from Human Feedback (RLHF) to the task of audio captioning, addressing a key limitation in current systems. Traditional audio captioning relies on supervised learning with expensive, manually curated audio-caption pairs, which may not reflect nuanced human preferences in real-world listening scenarios. The proposed method, detailed in a paper submitted to Interspeech 2026, introduces a scalable alternative that aligns AI-generated audio descriptions more closely with what humans find natural and correct, particularly when baseline models fail.
The technical core of the framework is a reward model based on Contrastive Language-Audio Pretraining (CLAP), which is trained on human-labeled pairwise preference data to capture subtle judgments about caption quality. This reward model is then integrated into a reinforcement learning loop to fine-tune any existing audio captioning system, eliminating the need for costly ground-truth annotations during the alignment phase. Extensive human evaluations across multiple datasets show the method produces captions significantly preferred over baseline models. Crucially, the framework achieves performance comparable to fully supervised approaches, demonstrating that effective alignment with human preferences is possible at scale, paving the way for more adaptable and user-centric audio AI applications.
- Uses RLHF to align audio captions with human preferences, moving beyond costly supervised learning with paired data.
- Trains a CLAP-based reward model on human-labeled pairwise preferences to judge caption quality.
- Fine-tunes baseline captioning systems without ground-truth data, achieving comparable performance to supervised methods.
Why It Matters
Enables scalable creation of more natural, user-preferred audio descriptions for accessibility, media, and AI assistants without expensive data labeling.