SIAgent: Spatial Interaction Agent via LLM-powered Eye-Hand Motion Intent Understanding in VR

A research team from unspecified institutions has introduced SIAgent (Spatial Interaction Agent), a novel framework that fundamentally rethinks how users interact in Virtual Reality. Instead of the traditional 'Operation-to-Intent' model—which forces users to memorize specific gestures like pinching or swiping—SIAgent implements an 'Intent-to-Operation' paradigm. This system allows users to interact through natural, intuitive eye and hand motions based on common sense, such as simply looking at an object and reaching for it. The core innovation is using a Large Language Model (LLM) to interpret these spatial movements, translating them into natural language to infer the user's goal, which is then executed by an AI agent.

The technical breakthrough is in its two-component architecture: intent recognition via LLM-powered translation of spatial data, and agent-based task execution. In user studies involving over 60 interaction tasks, SIAgent achieved a 97.2% intent recognition accuracy, surpassing the 93.1% accuracy of conventional gaze-plus-pinch techniques. Crucially, it also reduced physical strain, lowering arm fatigue and significantly boosting user preference and usability. The research validates the function of both eye gaze and hand motion channels for intent recognition and promises to make VR interfaces more accessible and intelligent. The team plans to release the source code and LLM prompts upon publication, paving the way for more natural, fatigue-free spatial computing.