Orchestra-o1 beats existing omnimodal agents by 10.3% with DA-GRPO

A new paper from Fan Zhang and colleagues presents Orchestra-o1, a framework designed to orchestrate multiple LLM-based agents across heterogeneous modalities—text, image, audio, and video. While existing agent swarms struggle to generalize beyond a narrow set of input types, Orchestra-o1 introduces a unified orchestration mechanism that dynamically decomposes tasks based on modality requirements, spawns specialized sub-agents on the fly, and executes subtasks in parallel. This design allows the system to tackle complex real-world tasks that involve diverse information sources simultaneously.

On the OmniGAIA benchmark, Orchestra-o1 achieves a 10.3% accuracy improvement over the next-best approach, demonstrating its effectiveness in omnimodal settings. To train the underlying 8B-parameter model, the researchers developed decision-aligned group relative policy optimization (DA-GRPO), an efficient agentic reinforcement learning technique. The resulting Orchestra-o1-8B model achieves state-of-the-art performance against all existing open-source omnimodal agents, marking a significant step toward practical multi-agent systems that seamlessly blend text, images, audio, and video understanding.