Seeing Eye to Eye: Enabling Cognitive Alignment Through Shared First-Person Perspective in Human-AI Collaboration

A research team led by Zhuyu Teng has developed Eye2Eye, a novel AR framework that addresses fundamental limitations in current AI assistants by establishing a shared first-person perspective between humans and AI. The system tackles two critical gaps identified in collaborative tasks: the communication gulf where users must translate parallel intentions into sequential verbal commands, and the understanding gulf where AI struggles with subtle embodied cues. By leveraging what the user sees through AR glasses as the primary communication channel, Eye2Eye creates cognitive alignment that traditional multimodal systems can't achieve.

Eye2Eye implements three core components that work in concert: joint attention coordination that allows the AI to follow and predict user focus points, revisable memory that maintains evolving common ground throughout tasks, and reflective feedback mechanisms that let users clarify and refine the AI's understanding in real-time. The team built an AR prototype and conducted both user studies and pipeline evaluations, with results showing significant improvements across key metrics compared to conventional voice-based assistants.

The framework represents a paradigm shift from command-based interaction to perspective-based collaboration, where the AI becomes an extension of the user's cognitive process rather than a separate tool requiring explicit instruction. This approach proved particularly effective in complex, dynamic tasks where intentions evolve rapidly and verbal descriptions become cumbersome. The research, accepted at ACM CHI 2026, demonstrates that first-person perspective sharing could become the next major interface breakthrough for human-AI collaboration.