MIT & UIUC: Internet videos boost robot training by 30%

A team of researchers from MIT and UIUC, led by Richard Li, explored a critical question: can ordinary internet videos of humans performing tasks be used to train robot manipulation policies? Currently, most human video datasets used for robot training are highly curated—featuring motions that mimic robot behavior and requiring specialized hardware for 3D hand tracking. The researchers argue that everyday videos are far more plentiful but suffer from a motion gap: the way humans move differs substantially from robot arms and grippers. To test this, they created a new dataset of 532 human videos totaling 28 hours, all with precise triangulated hand labels to ensure high-quality pose data.

Their experiments revealed two key insights. First, the quality of hand pose annotations directly impacts transfer performance—better hand tracking leads to better robot learning. Second, even with perfect hand poses, the inherent motion gap between human and robot movements hinders learning unless the vision and policy networks are specialized for each embodiment (e.g., a robot arm vs. a human hand). By implementing a cotraining recipe that accounts for these factors, the team achieved a striking 29.7% absolute improvement in success rate across six different manipulation tasks, particularly when robot training data was scarce. This work demonstrates a practical path to leveraging vast amounts of everyday internet video for robotic skill acquisition.