SignVLA: A Gloss-Free Vision-Language-Action Framework for Real-Time Sign Language-Guided Robotic Manipulation

A research team led by Xinyu Tan and Ningwei Bai has introduced SignVLA, the first gloss-free Vision-Language-Action framework that enables real-time robotic manipulation through sign language. Unlike conventional approaches that require labor-intensive gloss annotations as intermediate supervision, this system directly maps visual sign gestures to semantic instructions, eliminating annotation costs and avoiding the information loss inherent in gloss representations. The framework focuses on alphabet-level finger-spelling for robotic control, providing a more reliable and interpretable interface than continuous sign language recognition systems, particularly in safety-critical embodied environments where precision matters.

The technical pipeline transforms continuous gesture streams into coherent language commands through geometric normalization, temporal smoothing, and lexical refinement, ensuring stable interaction. The system is designed for future integration with transformer-based gloss-free sign language models, enabling scalable word-level and sentence-level semantic understanding. Experimental results demonstrate effective grounding of sign-derived instructions into precise robotic actions across diverse scenarios. This research represents a significant step toward accessible, scalable multimodal embodied intelligence that could transform how humans interact with robotic systems, particularly benefiting deaf and hard-of-hearing communities while advancing natural human-robot collaboration.