Ablation Study of Multimodal Perception, Language Grounding, and Control for Human-Robot Interaction in an Object Detection and Grasping Task

Researchers Zi Tian and Guanting Shen published an ablation study (arXiv:2605.00963) aiming to isolate the contributions of three key modules in a multimodal human-robot interaction pipeline for object detection and grasping: the large language model used for action extraction, the perception system for visual grounding, and the controller for motion execution. Rather than redesigning the full system, they systematically compared 3 LLMs, 5 perception configurations (including different visual grounding backbones), and 3 controllers under a common experimental protocol. The goal was to identify which components primarily drive execution time and which drive success rate, providing engineering guidance for future iterations.

In the first stage, each module was tested independently. Then the top performers were combined in a factorial design for end-to-end evaluation. Preliminary results show that the choice of LLM significantly impacts action extraction accuracy and speed, while perception configuration heavily influences visual grounding precision. The controller choice affects motion smoothness and grasp success. The study found that the largest gains come from improving the perception module, whereas LLM upgrades yield diminishing returns above a certain threshold. This paper offers concrete tradeoffs for teams building real-world robotic assistants.