Impact of Multimodal and Conversational AI on Learning Outcomes and Experience

A new study from Georgia Tech researchers provides crucial evidence on how to design effective AI tutors. In a randomized controlled trial with 124 participants learning biology, the team compared three learning interfaces: MuDoC, a document-grounded conversational AI that interleaves text and image responses; TexDoC, a similar system with text-only responses; and DocSearch, a traditional textbook interface with semantic search. The results were striking. Learners using the multimodal MuDoC system achieved the highest post-test scores and reported the most positive learning experience.

However, the study uncovered a critical and counterintuitive finding. While participants rated the text-only TexDoC chatbot as significantly more engaging and easier to use than the traditional DocSearch, it actually led to the lowest post-test scores. This reveals a dangerous disconnect where a more conversational, engaging interface can inflate a student's perceived understanding without improving actual learning. The researchers interpret these findings through Cognitive Load Theory, suggesting conversationality reduces extraneous mental load, but only when paired with multimodality (integrating visuals and text) does it increase the beneficial 'germane' load that leads to deeper learning. Without visuals, reduced cognitive effort may simply lead to overconfidence.