GenVid2Robot turns AI-generated videos into real robot actions with geometric consistency checks

Generated videos offer rich visual motion priors for robot manipulation, but they lack metric geometry, grasp grounding, and kinematic feasibility—making direct trajectory replay unreliable. GenVid2Robot, presented by Haohui Huang, Xi Yuan, and colleagues, solves this by introducing a rigid-geometric consistency framework. Given an initial RGB-D observation and a task instruction, it samples task-relevant semantic anchors from the real first frame, tracks them through generated video candidates, and verifies whether the 2D motion can be explained by first-frame RGB-D anchors under a sparse relative SE(3) model. This approach treats generated videos as uncertain visual motion hypotheses rather than direct demonstrations, only transferring geometrically consistent motion to the robot.

The accepted relative motion is then applied to a grasp-conditioned execution trajectory. A mask-constrained grasping module selects the optimal grasp-time TCP pose, ensuring the trajectory aligns with both the visual motion prior and the physical grasp configuration. To handle real-world imperfections like RGB-D noise, calibration residuals, and small contact-induced displacements, a bounded depth-compensation module corrects local depth-direction errors without requiring full online replanning. Real-robot experiments demonstrate that GenVid2Robot significantly improves the reliability of generated-video-guided manipulation by grounding visual priors with sparse metric geometry, grasp constraints, robot feasibility checks, and bounded execution feedback.