ACE-Ego-0: Researchers teach robots from human egocentric video data

Scaling robot training data is expensive and labor-intensive. Researchers from multiple institutions (including authors Hao Li, Ganlong Zhao, et al.) present ACE-Ego-0, a Vision-Language-Action (VLA) pretraining framework that leverages large-scale egocentric human video data to augment robot demonstrations. The key innovation is a scalable egocentric video-to-action pipeline that converts raw human videos into robot-format pseudo-action trajectories, making them directly comparable with robot data.

To bridge the divergence between human and robot action spaces, ACE-Ego-0 uses a unified action representation based on camera-space actions, morphology conditioning, and time-aligned action chunking. A reliability-aware training objective with a human auxiliary loss concentrates supervision on the most reliable signals from noisy human video data. The framework is instantiated on 4.53K hours of robot and simulation data plus 1.48K hours of egocentric human video.

Results show that incorporating large-scale human supervision under this reliability-aware weighting consistently improves both unified joint pretraining and supervised fine-tuning. ACE-Ego-0 achieves state-of-the-art performance on RoboCasa GR1 TableTop and RoboTwin 2.0 benchmarks, and demonstrates strong zero-shot transfer to real-world bimanual manipulation tasks. This work opens a path to massively scalable robot learning by tapping into the wealth of existing human egocentric video.