GigaWorld-1 Reveals Roadmap for Robot Evaluation with 324K Rollouts
324,000 simulated rollouts show world models need long-horizon consistency over visual flashiness.
Evaluating embodied robot foundation models is a critical bottleneck—real-world rollouts are slow, costly, and human-supervised. The GigaWorld Team tackles this with WMBench, a benchmark built from real-robot teleoperation data and matched policy rollouts covering diverse manipulation tasks. They analyze 7 video world models, 4 action representation schemes, and over 324,000 simulated policy rollouts paired with real robot executions, enriched by community submissions from the CVPR 2026 GigaBrain Challenge and 12,000+ hours of training videos.
Their core insights: evaluator quality is dominated by long-horizon, action-faithful rollout consistency, not short-term visual realism. Pretraining gains come from balancing general world knowledge with robot-specific controllability. Architectural choices—action encoding, memory design, and evaluator-focused post-training—strongly determine alignment with real behavior. Based on these findings, they built GigaWorld-1, a world model specifically optimized for policy evaluation, and fully released all code, models, datasets, and toolkits to advance scalable evaluation research.
- WMBench benchmark covers diverse manipulation tasks with real-robot teleoperation data and matched policy rollouts.
- Key insight: long-horizon, action-faithful rollout consistency dominates evaluator quality over short-term visual realism.
- GigaWorld-1 optimizes world models for policy evaluation using action encoding, memory design, and post-training; all assets released.
Why It Matters
Accelerates evaluation of robot foundation models, reducing reliance on slow, costly real-world testing.