PrismAudio By Qwen: Video-to-Audio Generation

Researchers from Qwen have introduced PrismAudio, a groundbreaking framework for video-to-audio (V2A) generation that tackles a core problem in the field: objective entanglement. Existing methods often conflate the four critical perceptual goals—semantic consistency, audio-visual synchrony, aesthetic quality, and spatial accuracy—into single, conflicting loss functions. PrismAudio solves this by decomposing the reasoning process into four specialized Chain-of-Thought (CoT) modules, each paired with a targeted Reinforcement Learning (RL) reward function. This 'CoT-reward correspondence' allows for multidimensional RL optimization, guiding the model to generate coherent reasoning across all perspectives simultaneously, which preserves interpretability while improving performance.

To make this complex RL training practical, the team developed Fast-GRPO, a novel method using hybrid ODE-SDE sampling that dramatically reduces computational overhead compared to standard GRPO. They also created AudioCanvas, a new, more rigorous benchmark designed to be distributionally balanced and cover diverse, challenging scenarios. It includes 300 single-event classes and 501 multi-event samples, providing a more realistic testbed than previous datasets like VGGSound. Experimental results show PrismAudio achieves state-of-the-art performance across all four perceptual dimensions on both in-domain and the new out-of-domain AudioCanvas benchmark, demonstrating robust generalization.