Speech2Grasp enables humanoid robots to grasp objects via speech commands
Robots now understand spoken commands to grasp objects without text transcription.
A team of researchers from multiple Vietnamese institutions has developed Speech2Grasp, a novel framework that enables humanoid robots to directly understand spoken commands for grasping objects. The work addresses a key limitation of current vision-language models (VLMs) like ALBEF, which typically rely on text input instead of the more natural speech. By adding a lightweight MLP-based projector, the team demonstrates a data-efficient method to transfer a well-established text-conditioned grasp detection model to speech inputs. This approach preserves the model's semantic discrimination and robustness while eliminating the need for large speech datasets or architectural overhauls.
In real-world experiments with a humanoid robot, Speech2Grasp outperformed the traditional cascaded ASR-based pipeline—where speech is first transcribed to text then processed—both in accuracy and inference speed. The direct speech-to-action approach reduces latency by bypassing the intermediate transcription step. The findings suggest a practical paradigm: rather than building speech-specific models from scratch, researchers can adapt powerful text-conditioned systems to speech with minimal cost. This could accelerate the development of natural, voice-controlled humanoid robots for homes, warehouses, and manufacturing.
- Uses ALBEF vision-language model as base, adapted via a lightweight MLP projector for speech input.
- Data-efficient: requires only minimal speech data (not thousands of hours) to transfer from text to speech.
- Real-world humanoid robot tests show it outperforms traditional ASR-based pipelines with lower inference latency.
Why It Matters
Enables more natural human-robot interaction by letting robots understand spoken commands directly without text transcription.