arXiv Study: AI Overviews in Academic Search Reduce Mental Load

A new arXiv paper (2607.03421) by Schott et al. examines AI-generated summaries of search engine results pages (SERPs) as a support feature in academic search for social science information. The study used a formative mixed-methods design: first, manually evaluating summaries of the top five results for 10 queries using one commercial and one open general-purpose model, producing a six-category error taxonomy and five safeguards for scholarly deployment. Then, a within-subjects user study with 30 participants compared interfaces with and without AI summaries.

Confirmatory analyses showed consistent but non-significant trends favoring AI summaries for subjective workload, perceived usefulness, satisfaction, and decision-making confidence. Exploratory analyses suggested lower mental demand and frustration. Behaviorally, participants rarely expanded summaries and made slightly fewer result clicks and query reformulations. Drawing on Information Foraging Theory, the authors propose that AI summaries concentrate SERP-level information scent to support early triage. The work concludes that these summaries are context- and user-dependent aids, not universal improvements, and contributes an error taxonomy, safeguard guidance, and design implications for scholarly search.