Gaze-Aware Task Progression Detection Framework for Human-Robot Interaction Using RGB Cameras
A new calibration-free system detects task completion with 77.6% accuracy using only a robot's built-in RGB camera.
A team of researchers from Vrije Universiteit Amsterdam has developed a novel framework that enables robots to detect human task progression using only a standard RGB camera, eliminating the need for specialized eye-tracking hardware. The system, detailed in a paper accepted for IEEE Robotics and Automation Letters, monitors a user's gaze across three defined Areas of Interest (AOI): a task interface (like a tablet), the robot's face, and elsewhere. It leverages the natural behavior where a user shifts their gaze from the task to the robot's face to signal completion, formalizing this into a detection model.
In validation using a 'First Day at Work' onboarding scenario, the framework demonstrated a task completion detection accuracy of 77.6%. While it exhibited slightly higher response latency than a traditional button-press method, it preserved information retention and led to significant improvements in user comfort, social presence, and perceived naturalness. Notably, most participants reported they did not consciously use eye movements to guide the interaction, highlighting the intuitive, non-invasive nature of the approach. This work proves the feasibility of creating more natural and engaging human-robot interactions using low-cost, existing hardware.
- Uses only a robot's built-in monocular RGB camera (640x480), removing dependency on expensive eye-tracking hardware.
- Achieved 77.6% task completion detection accuracy in a validated scenario, balancing accuracy with interaction latency.
- Significantly improved user metrics vs. button-based interaction: better comfort, social presence, and perceived naturalness.
Why It Matters
Enables more intuitive, low-cost human-robot collaboration for assistive robots, customer service, and industrial training without specialized hardware.