Researchers propose SMC-ITA for better video-to-audio sync

Researchers from the University of Tokyo have introduced SMC-ITA (Sequential Monte Carlo Inference-Time Alignment), a novel approach to improve video-to-audio (V2A) generation by addressing alignment challenges during inference. Unlike prior methods that focus on model architectures or training objectives, SMC-ITA treats alignment as a search problem, dynamically reallocating computational resources using multi-dimensional cross-modal rewards.

The method combines two key components: lookahead-based reward estimation and sequential Monte Carlo resampling. Lookahead improves the reliability of intermediate reward estimates, while systematic resampling enhances synchronization between video frames and generated audio. Under matched neural function evaluation (NFE) budgets, SMC-ITA achieves a 55.67% reduction in DeSync errors, a 20.23% improvement in IB-score (semantic consistency), and a 15.44% boost in audio quality compared to baseline methods like Best-of-N and Beam Search.