Self-supervised robot learning cuts calibration, hits 66% grasp success on UR5e

A team led by Andres Rivas and colleagues from UFSM published a paper (arXiv:2608.07553) presenting a self-supervised approach for visual robotic manipulation that eliminates the need for human demonstrations, object-specific programming, or explicit camera-to-robot calibration. The robot automatically generates demonstration data by moving around a target pose while capturing wrist-mounted RGB images. These images are paired with pose labels collected via ROS 2 and simulated in Isaac Sim, and a convolutional network learns to regress relative translation and rotation from single-frame observations. The pipeline separates datasets for planar refinement and coarse 3D approach, using a coarse-to-fine controller during execution.

In simulation, the refinement stage reduced final planar dispersion from 9.69 mm to 5.38 mm. Real-world tests on a UR5e robot equipped with a gripper and monocular camera achieved end-to-end grasp success rates of 66.6% and 63.6% for two objects without rotation, while maintaining partial robustness under rotated conditions. The results demonstrate that automatically generated demonstrations can support practical visual manipulation with minimal setup effort, though challenges remain in depth prediction and object-dependent generalization. The paper was accepted at ICCAS 2026 and highlights a promising path toward autonomy in robotic learning.