LLM-Guided Reinforcement Learning Enhances Audio-Visual Speech Enhancement
New method uses LLM feedback as reward to train speech enhancement models.
Deep Dive
A reinforcement learning framework for Audio-Visual Speech Enhancement (AVSE) uses an LLM to generate natural language descriptions of enhanced speech. A sentiment model converts these into 1-5 ratings as PPO rewards. On the AVSEC-4 dataset, the method outperforms a supervised baseline and a DNSMOS-based RL baseline in PESQ, STOI, neural quality metrics, and subjective listening tests. Accepted at Interspeech 2026.
Key Points
- Uses an LLM to generate natural language descriptions of enhanced speech, converted via sentiment analysis into a 1-5 PPO reward.
- Outperforms supervised baselines and DNSMOS-based RL on AVSEC-4 dataset across PESQ, STOI, and subjective tests.
- Accepted at Interspeech 2026; authored by researchers from Academia Sinica and UC Irvine.
Why It Matters
LLM-based interpretable rewards could make AI speech enhancement more aligned with human perception.