Researchers build hybrid attention pipeline for robotic heads to read human cues

A team including Pablo Moraes, Monica Rodriguez, and Christopher Peters has proposed a hybrid visual attention estimation pipeline designed to make human-robot interaction (HRI) more natural. Built on the open-source InMoov robotic head ecosystem, the system combines two complementary perception layers: a fast geometric layer that tracks facial features and head pose at high frequency, and a semantic layer that uses a vision-language model to analyze raw egocentric camera frames. Together, they produce contextual attention labels indicating whether the user is focused on the robot, using their phone, or looking elsewhere.

The two signals feed into a finite state machine that regulates the robot's behavior, enabling adaptive actions such as initiating interaction, waiting, resuming a paused conversation, or returning to a rest state. The researchers evaluated the system with 10 participants across 40 trials, comparing baseline and adaptive interaction conditions. Results showed reliable interaction start across all trials, consistent pause behavior when the user was distracted, and confirmed that the geometric and semantic outputs provided non-redundant, complementary information. This hybrid approach offers a practical path for robots to sense and adapt to human attention in real time.