Microsoft study: ChatGPT users learn less than Google searchers

A new study from Microsoft Research, published on arXiv, investigates how generative AI tools like ChatGPT impact information seeking and learning. Researchers conducted an 8-day between-subjects field experiment where participants pursued informal learning using either ChatGPT or Google Search. Using a daily diary protocol, they gathered in-situ data on participants' information-seeking processes. The study aimed to understand the implications of offloading cognitive tasks to AI, particularly for knowledge acquisition and critical thinking.

The findings reveal significant drawbacks in the ChatGPT group. Participants experienced diminished agency as they offloaded information selection to AI, which paradoxically increased their metacognitive load—the mental effort required to monitor and control their own learning. Two sources of distortion emerged: first, ChatGPT outputs are biased towards providing solution-oriented artifacts over principled knowledge; second, the conversational interaction paradigm systematically reduces exploration of the broader knowledge space. As a result, ChatGPT users had worse learning outcomes on average than Google users, especially for higher-order critical learning. The work suggests inherent tensions between convenience and meaningful learning, with broader implications for understanding AI's risks to human cognition.