AI study uses robots to measure human rapport in stores
Study finds Gemini 2.5 Flash best at detecting real-world social cues in retail robots
Researchers from Japanese institutions have published findings on automatically estimating rapport scores in real-world human-robot interactions (HRI). The team analyzed 62 multimodal recordings from a Japanese drugstore to evaluate how well AI could assess third-party-rated social connection quality in unstructured environments where users might disengage or form groups naturally.
The study compared various approaches including zero-shot LLMs, pretrained text models, and specialized audio-visual models. Key results showed Google's Gemini 2.5 Flash performing strongly as a standalone model, while combining it with Facebook's HuBERT (audio) and Meta's V-JEPA (visual) models achieved the highest accuracy. Performance varied significantly based on interaction duration and group size, highlighting the need for context-aware evaluation systems rather than lab-based benchmarks.
The research was accepted at ACM's ICMI 2026 conference and represents a significant step toward autonomous robot behavior adaptation in real-world settings.
- Evaluated on 62 real-world HRI sessions from a Japanese drugstore with multimodal recordings
- Gemini 2.5 Flash achieved strong standalone performance, while HuBERT + V-JEPA fusion performed best overall
- Performance varied with interaction duration and group size, suggesting context matters more than lab conditions
Why It Matters
This research enables robots to dynamically assess and adapt to human social cues in real environments, improving autonomous interaction capabilities.