Utilizing Inpainting for Keypoint Detection for Vision-Based Control of Robotic Manipulators

A team of researchers has published a novel framework for vision-based robotic control that cleverly uses AI inpainting to solve a major data-labeling problem. Their goal is to enable robots to be controlled using only their natural visual features, eliminating the need for external markers like ArUco codes, which are often impractical. The core innovation is a two-stage inpainting process. First, during training, they attach markers to a robot, record images, and then use an inpainting model to digitally erase the markers and reconstruct the occluded robot surface. This creates a perfectly labeled dataset of "natural" robot images without manual annotation, camera calibration, or a precise robot model.

For runtime operation, the system employs a second, real-time inpainting model to handle partial occlusions—like a human hand blocking the view—ensuring continuous keypoint detection. These detections are further refined by an Unscented Kalman Filter (UKF) for smooth and stable predictions. The result is a fully model-free control strategy where a robot can servo to a desired configuration using only a camera feed of its own body. The paper demonstrates successful control under both full visibility and partial occlusion, proving the robustness of the approach.

This method represents a significant shift from traditional marker-based or model-dependent visual servoing. By leveraging generative AI (inpainting) for both data synthesis and real-time perception, it opens the door for more flexible and deployable robots in unstructured environments where placing markers is impossible. The technique effectively bridges the sim-to-real gap for vision-based control by creating realistic, labeled training data directly from the physical world.