Frustrometer uses mouse and gaze to detect when users are stuck in visualizations

A team of researchers from Austria, the US, and Europe developed the Frustrometer, a convolutional neural network (CNN) that detects when users are stuck or frustrated while interacting with data visualization dashboards. In a controlled study with 14 participants performing analytical tasks on two interactive dashboards, they captured a wide range of signals: eye movement, pupil dilation, galvanic skin response, heart rate, head orientation, mouse dynamics, and keyboard events. Participants also self-assessed their performance, and researchers annotated moments of stuckness. The goal was to solve the timing problem of when to offer help—too early disrupts flow, too late users have already gone astray.

The results revealed that autonomous physiological responses such as heart rate and galvanic skin response provided limited insight into frustration levels, and head orientation was also poorly correlated. Remarkably, mouse movements alone showed strong predictive power for some participants, and gaze data (eye tracking) was similarly effective. This implies that lightweight instrumentation—just tracking mouse behavior—could be sufficient for real-time frustration detection in practical systems. The paper concludes by discussing how these findings can inform adaptive guidance systems that use a multimodal approach (combining mouse and gaze) to offer timely, non-intrusive help to users performing complex visualization tasks.