GLANCE: A Global-Local Coordination Multi-Agent Framework for Music-Grounded Non-Linear Video Editing
A new multi-agent framework uses a 'bi-loop' architecture to automatically create music-synced video mashups from large clip libraries.
A research team from Tsinghua University, the University of Illinois Urbana-Champaign, and other institutions has introduced GLANCE, a novel multi-agent framework designed to automate the complex task of creating music-synced video mashups. The system tackles the challenge of non-linear video editing, where it must select and sequence clips from large, heterogeneous source libraries to form a coherent narrative that aligns with musical rhythm, user intent, and long-range structural constraints. Unlike previous fixed pipelines, GLANCE employs a sophisticated 'bi-loop' architecture: an outer loop for long-horizon planning and task-graph construction, and an inner loop where specialized agents follow an 'Observe-Think-Act-Verify' process to execute and refine segment-level edits.
To resolve conflicts that arise when combining individual video segments, the framework features a dedicated global-local coordination mechanism. This includes a context controller, a conflict region decomposition module, and a bottom-up dynamic negotiation process between agents. For evaluation, the team created MVEBench, a new benchmark that categorizes editing difficulty, and used an 'agent-as-a-judge' framework for scalable assessment. When powered by the GPT-4o-mini model, GLANCE demonstrated significant performance gains, improving over the strongest existing baseline by 33.2% and 15.6% on different task settings. Human evaluations further confirmed the quality of its generated videos, validating both the system's output and its novel evaluation methodology.
- Uses a 'bi-loop' architecture with an outer planning loop and an inner 'Observe-Think-Act-Verify' editing loop for agents.
- Introduced a global-local coordination mechanism with conflict resolution to manage cross-segment inconsistencies in the final video.
- Outperformed the strongest baseline by 33.2% using GPT-4o-mini and was validated with a new benchmark, MVEBench, and human evaluation.
Why It Matters
Automates complex, creative video editing tasks, potentially revolutionizing content creation for social media, marketing, and entertainment by syncing visuals to music.