Sirotkina's study: AI saliency models beat by simple center, biased by demographics

Computer vision saliency models promise to predict where people look, powering a billion-dollar attention-prediction industry. But according to a new arXiv paper by Elena Sirotkina, those models barely beat a naive baseline and carry hidden demographic biases. She tested leading saliency networks against 11.4 million real webcam gaze points collected from 3,023 US adults — recruited to national quotas — while they viewed circulating news photographs. The result: an untrained central marker (simply always predicting the image center) outperformed every trained network. The models' added content falls exactly where these audiences never look.

The study also exposes systematic bias in whatever accuracy remains. The models' predictions align better with younger, White, and politically moderate viewers, while performing worse for older, Black, and ideologically extreme participants. To fix this, Sirotkina introduces a new evaluative framework built on what a group's own gaze reveals about whether a model can learn that group at all. She applies it across demographic axes and argues that systems deciding what people see can learn to see everyone — but only if the industry adopts rigorous, gaze-based benchmarks. This paper supplies the standard for judging such claims.