RoboSnap turns single photos into interactive robot simulation scenes instantly

RoboSnap, developed by a team including Shujie Zhang and Chunhua Shen, addresses a critical bottleneck in robotics: the slow and expensive process of building realistic simulation environments from real-world spaces. The framework takes just one RGB image and produces a fully interactive scene ready for robot learning and evaluation. Its key innovation is a layered architecture that separates the physics-critical interaction area from the surrounding visual context. Foreground objects are refined with collision-aware processing to ensure stable robot interactions, while background appearance is preserved using 3D Gaussian splatting, enabling novel-view rendering without compromising physical accuracy.

The system was validated extensively on scenes from the DROID dataset (564 real-world scenes) and on live real-robot tasks. RoboSnap achieves reliable trajectory replay, supports task-specific synthetic data generation for policy training, and demonstrates meaningful sim-to-real correlation for policy evaluation. The authors also release DROID-Sim, a companion real-to-sim dataset built from those 564 scenes, to accelerate further research. Their experiments suggest that the true value of real-to-sim methods lies not just in visual fidelity but in turning real environments into reusable infrastructure for robot learning.