VR + AI training for autistic workers: grasp interactions boost engagement

A new study from researchers Ziming Li and Roshan Peiris investigates how generative AI can enhance VR-based communication training for autistic individuals. The paper, presented at ASSETS 2026, focused on workplace communication, where soft skills are embedded in ongoing tasks and environment context, not just isolated conversation. The team built on prior work using LLM-driven agents that generate dialogue grounded in VR environments and user interactions. They designed an exploratory study with 9 autistic trainees and 7 job coaches comparing three modalities: conversation-only (C), conversation with environmental objects (C+O), and conversation with objects plus grasp interactions (C+O+G). The goal was to understand how different levels of environmental grounding influence the training experience.

Results showed that usability and workload were comparable across all three conditions, but both trainees and coaches strongly preferred the most interactive, fully grounded condition (C+O+G). Participants reported that grasping objects while conversing helped maintain engagement and made communication practice more natural, integrating it directly into task performance. The researchers discuss design considerations for creating flexible, interactive VR training that mirrors real workplace demands. This approach could improve soft skills development for autistic individuals, addressing barriers to employment and social integration. The study highlights the potential of combining generative AI, VR, and embodied interaction for inclusive training tools.