Large Language Models in Teaching and Learning: Reflections on Implementing an AI Chatbot in Higher Education

A research team from the Technical University of Denmark, led by Fiammetta Caccavale, conducted a pioneering study on implementing a specialized AI assistant in higher education. To mitigate the known risks of large language models (LLMs)—such as generating incorrect 'hallucinations' and lacking domain-specific knowledge—the team built a chatbot using a Retrieval-Augmented Generation (RAG) model. This approach grounds the AI's responses in a curated knowledge base. The assistant was deployed to replicate a previously teacher-led, time-intensive exercise in a university-level course, creating a direct comparison point between human and AI-led instruction.

The study's methodology was rigorous, featuring three separate experiments that used iterative, mixed-methods approaches including a crossover design. This allowed the researchers to collect robust data on central questions: how student motivation was affected, how students perceived the quality of AI-generated responses versus human teaching, and the actual impact on academic performance. The findings offer one of the first controlled, empirical looks into the practical integration of LLMs in specialized academic settings, moving beyond theoretical discussion.

The results provide direct insights into the pedagogical feasibility of such tools. The paper discusses the specific challenges and opportunities identified, offering a roadmap for educators and institutions looking to responsibly embed AI into their curricula. This work is significant as it moves the conversation from speculative potential to evidence-based implementation, addressing core concerns about accuracy and effectiveness in the high-stakes environment of university education.