Task-Aware Bimanual Affordance Prediction via VLM-Guided Semantic-Geometric Reasoning

A team from TU Darmstadt and the Max Planck Institute for Intelligent Systems has published a paper introducing a novel framework for task-aware bimanual robot manipulation. The core innovation is reframing the problem as a joint affordance localization and arm allocation challenge. Their system first builds a consistent 3D scene representation from multi-view RGB-D observations and generates potential 6-DoF grasp candidates. It then uses a pre-trained Vision-Language Model (VLM) as a semantic filter, querying it to identify task-relevant contact regions on objects and to logically assign each task to a specific robot arm. This fusion of geometric validity with high-level task understanding is what sets it apart from prior methods.

The approach was rigorously evaluated on a real dual-arm robot platform across nine distinct tasks spanning four categories: parallel manipulation (e.g., carrying a tray), coordinated stabilization (holding an object steady), tool use, and human handover. The results demonstrated consistently higher task success rates compared to baselines that relied solely on geometry or coarse semantic segmentation. Crucially, the method generalizes across object categories and task descriptions without requiring category-specific training, as the VLM provides the necessary semantic grounding. This represents a significant step toward enabling reliable, two-handed manipulation in unstructured, real-world environments where a robot must understand not just how to grip, but why and with which 'hand'.