Visual-RRT: Finding Paths toward Visual-Goals via Differentiable Rendering

A team from KAIST has introduced Visual-RRT (vRRT), a breakthrough motion planning algorithm that allows robots to navigate toward goals defined purely by visual inputs, such as a target image or a demonstration video. Traditional Rapidly-exploring Random Tree (RRT) planners require precise numerical coordinates for a goal, like specific joint angles, which are often unavailable in real-world scenarios where instructions are visual. vRRT solves this by merging two core techniques: the robust, exploration-focused sampling of RRTs with gradient-based optimization from differentiable rendering. This hybrid approach lets the robot 'see' a goal state and computationally work backward to find a feasible motion path to achieve it.

The system employs two key innovations to make this vision-to-action pipeline efficient. First, a frontier-based strategy dynamically balances between exploring new areas and exploiting visually promising regions identified by the renderer. Second, an inertial gradient expansion method maintains optimization momentum across different branches of the search tree, preventing wasted computation. In extensive experiments, vRRT successfully enabled robots like the Franka Panda, UR5e, and Fetch manipulator to plan and execute movements based on visual goals, effectively bridging the gap between high-level visual perception and low-level motion control. The code is publicly available, paving the way for more intuitive robot programming and teleoperation.