Neural network-based encoding in free-viewing fMRI with gaze-aware models

A research team led by Dora Gozukara and Nasir Ahmad has developed a breakthrough AI model that dramatically improves how we study the brain's visual system. Their 'gaze-aware encoding model' combines convolutional neural network (CNN) features with real-time eye-tracking data to predict fMRI brain activity during natural viewing conditions. Traditional methods force participants to stare at a fixed point while watching stimuli, which suppresses natural visual processing and adds cognitive load. This new approach instead tracks where people actually look, sampling only the locally relevant parts of visual scenes that correspond to each fixation.

Trained on the StudyForrest dataset featuring task-free movie viewing, the model achieved remarkable efficiency: it matched the performance of conventional encoding models while using 112x fewer parameters. This represents a shift from billion-parameter models to million-parameter models without sacrificing accuracy. The approach proved particularly effective for participants with more dynamic eye-movement patterns, suggesting it better captures individual differences in visual attention.

This breakthrough enables more ecologically valid neuroscience research by allowing studies in truly naturalistic settings—people can watch movies, play games, or navigate virtual environments without artificial constraints. The model's efficiency also makes it practical for broader applications, potentially accelerating brain-computer interface development and improving our understanding of how the brain processes complex, real-world visual information. By bridging the gap between controlled laboratory conditions and natural human behavior, this research opens new possibilities for studying cognition in authentic contexts.