A Model-based Visual Contact Localization and Force Sensing System for Compliant Robotic Grippers

A team led by Kaiwen Zuo has introduced a model-based visual contact localization and force sensing system for compliant robotic grippers. The system addresses the challenge of estimating grasp force without dedicated force sensors, which is critical for manipulating delicate objects. Unlike typical end-to-end deep learning approaches that struggle to generalize, this method combines structural key point extraction from RGB-D wrist camera images with an inverse finite element analysis simulation in the Simulation Open Framework Architecture (SOFA). The iterative contact localization sub-system uses a deep learning-based online 3D reconstruction and pose estimation pipeline to dynamically update contact location, making it robust to visual occlusion and unseen objects. This model-based approach is tailored for fin-ray-shaped soft grippers.

The system demonstrated impressive accuracy: an average root mean square error (RMSE) of 0.23 N and normalized root mean square deviation (NRMSD) of 2.11% during the load phase, and 0.48 N and 4.34% over the entire grasping process. These results held across interactions with various objects under different conditions, showcasing generalization capability. The 8-page paper, published in IEEE Robotics and Automation Letters, highlights the potential for real-time model-based indirect force sensing. By leveraging existing wrist cameras commonly used for robot control, the approach avoids added hardware cost and complexity, making it accessible for industrial and research applications where precise force feedback is needed for safe object handling.