Transparent Fragments Contour Estimation via Visual-Tactile Fusion for Autonomous Reassembly

A research team led by Qihao Lin has published a novel AI framework that tackles the challenging problem of autonomously reassembling broken transparent objects like glass, optical lenses, or cultural artifacts. The system, detailed in the paper "Transparent Fragments Contour Estimation via Visual-Tactile Fusion for Autonomous Reassembly," addresses a major robotics hurdle: transparent fragments are notoriously difficult for standard computer vision due to their strict optical properties, irregular shapes, and hard-to-detect edges.

To solve this, the researchers propose a multi-sensory approach that mimics how humans handle fragile pieces. First, a visual network called TransFragNet identifies and segments potential grasping positions on a fragment. A two-finger gripper, equipped with Gelsight Mini tactile sensors, then picks up the piece. As it holds the fragment, the sensors reconstruct a detailed 3D map of the lateral edge through touch. This crucial tactile data is fused with the initial visual cues in a material classifier, creating a complete and accurate contour estimation that vision alone could not achieve.

The team also built and released a significant new resource to advance the field: the TransFrag27K dataset. This includes a scalable synthetic data generation pipeline and over 27,000 multiscene examples of broken fragments from various transparent objects. The framework is completed with a contour matching and reassembly algorithm that uses multi-dimensional similarity metrics, providing a reproducible benchmark for future research. The experimental validation shows the system's effectiveness in real-world scenarios, paving the way for robots that can perform delicate restoration and repair tasks autonomously.