Robots Learn Real-World Tasks by Watching 120K Hours of Video

For a long time, teaching a robot to do something—say, open a drawer or flip a pancake—meant collecting countless examples of exactly that action, often with a real robot moving step by step. This is incredibly slow and expensive. But humans don't learn that way. We watch other people, pick up patterns, and try things ourselves. ZimaBlue, a new research system from a team of scientists, brings that same logic to machines: instead of only feeding them robot demonstrations, it trains them on massive amounts of regular egocentric video—footage shot from a person's point of view, showing hands picking things up, using tools, and cooking meals.

The core idea is a three-stage learning process. First, ZimaBlue watches large-scale videos of humans and robots to understand how objects behave and move in the real world. Then it links that visual understanding to actual robot actions using a smaller set of robot data. Finally, it adapts to a specific robot so it can be deployed. Think of it like learning a new sport: you watch plenty of game footage first, then you practice with a coach, then you play in real games. This "video pre-training" lets the robot absorb vastly more experience than any laboratory could physically demonstrate, because a camera can record anywhere, anytime.

The numbers are striking. When ZimaBlue was tested on real robots doing tasks they had never encountered, its success rate jumped from 36.1% to 77.8% after training on over 120,000 hours of embodied video. For reference, that's nearly 14 years of nonstop footage. It also runs quickly enough for real-time use, making 30 action predictions per second on an NVIDIA RTX 4090—a high-end graphics card you can buy at a computer store. That speed matters because a robot that hesitates isn't useful in a kitchen or workshop.

There are still big hurdles ahead. Robots in the lab are not yet home helpers, and learning from video won't automatically solve complex social or dangerous situations. But this approach shows that robots can get dramatically more capable without needing endless hand-coded instructions or custom demonstrations. As video data becomes even easier to collect, robots could soon learn flexible, common-sense skills the same way we do—by watching the world around them.