New monocular vision framework grasps soft and rigid objects without tactile sensors

Researchers Shail Jadav and Dongheui Lee have developed a monocular vision-based control framework for robotic grasping that unifies handling of both soft and rigid objects within a single pipeline. Unlike existing approaches that require separate strategies—often relying on tactile sensing, object-specific models, or specialized grippers—this system uses only RGB camera input and a standard position-controlled gripper. The framework integrates multiple computer vision components: open-vocabulary object detection, image segmentation, boundary-aware point assignment, real-time point tracking, and monocular depth estimation. A key innovation is a language-based stiffness estimation model that infers an object's expected compliance from its semantic description (e.g., 'lettuce' suggests high deformability) and provides an object-level prior for selecting the grasping strategy before contact.

For deformable objects, grasp adaptation is governed by a Procrustes-based dissimilarity measure computed from tracked keypoints, which acts as a visual proxy for deformation. For rigid objects, gripper width is regulated through the scaling of tracked point distances. The method was validated in real-world pick-and-place experiments on a Franka Emika Research 3 arm with objects having substantially different mechanical properties: lettuce, fresh mozzarella cheese, croissants, paper towels, and hard plastic bottles. Results demonstrate stable grasping across both categories using only visual feedback, highlighting a practical, sensor-efficient, and generalizable approach for food handling and household manipulation. The paper is accepted at IEEE/ASME AIM 2026.