NEST-V1 translates Nepali speech into emotion-driven sign avatars

NEST-V1 (Nepali Emotion and Speech Transformer - Version 1), unveiled in a pilot study on arXiv by researchers Jatin Bhusal and Salma Tamang, marks a significant step toward emotion-aware sign language translation for low-resource languages. The framework takes spoken Nepali words and generates avatars performing corresponding sign language gestures conditioned on detected emotional states (happy, neutral, sad). Using a shared acoustic encoder, NEST-V1 simultaneously performs Automatic Speech Recognition (ASR) and emotion classification—a departure from traditional systems that handle these tasks separately. The model is remarkably lightweight at just 22.1 million parameters, making it suitable for edge deployment on mobile devices or low-powered hardware. Currently, the system is trained on a dataset of 600 labeled audio samples from 50 speakers for four common Nepali words: “thank you”, “hello”, “house”, and “me.”

Despite its limited vocabulary, NEST-V1 delivers strong preliminary results: 81.1% ASR accuracy and 79.21% emotion recognition accuracy. More impressively, its shared encoder architecture achieves 37% parameter efficiency compared to separate models performing the same tasks. This efficiency is critical for real-time applications and for scaling to larger vocabularies in future iterations. The research opens a clear pathway to building inclusive communication tools for the hearing-impaired community in Nepal, where sign language resources are scarce. By integrating emotional expression into avatars, NEST-V1 promises more natural, empathetic interactions—a feature often missing from current sign language translation systems.