HiFiGaze: Improving Eye Tracking Accuracy Using Screen Content Knowledge
A new system leverages the 4K camera on your laptop to read the screen's reflection in your eye.
A team from Carnegie Mellon University, led by Taejun Kim, has introduced HiFiGaze, a research project that rethinks gaze estimation for consumer devices. The core innovation leverages a simple but powerful fact: modern smartphones, laptops, and desktops have high-quality front-facing cameras (4K or greater) and, crucially, they know exactly what is being shown on their own screens. HiFiGaze captures the 2D reflection of this screen content in the user's cornea. By using its precise knowledge of the displayed image, the system can robustly segment this reflection, whose location and size directly encode where the user is looking.
In a user study, their best-performing model demonstrated an ~8% reduction in mean tracking error over traditional appearance-based models that only analyze the eye's appearance. A supplemental finding revealed that positioning the gaze-tracking camera at the bottom of the device, rather than the top, could yield an additional 10-20% improvement in accuracy. This work, accepted for ACM CHI 2026, shifts the paradigm from trying to infer gaze from the eye alone to using the device's internal state as a key signal, paving the way for software-based, high-fidelity eye tracking without specialized hardware.
- Leverages device's knowledge of its own screen content to analyze corneal reflections for gaze tracking.
- Reduces mean tracking error by ~8% compared to standard appearance-based models in initial studies.
- Camera placement matters: positioning at the bottom of the device can improve accuracy by 10-20%.
Why It Matters
Enables highly accurate, software-based eye tracking on existing consumer devices, unlocking new accessibility and interaction paradigms.