AI Tutor LEA Shows Faithfulness Drop When Teaching Unfamiliar Courses
First classroom test of LEA reveals AI accuracy fades with curriculum distance...
A new study from researchers at the University of Abertay (Teri Rumble, Javad Zarrin, P. George Lovell, Ruth Falconer) extends their earlier work on the Learning Engagement Assistant (LEA), an adaptive AI tutoring agent that combines Retrieval-Augmented Generation (RAG) with structured Knowledge Component (KC) models. Previous validation used only synthetic learner agents on a single STEM course (CMP511). This paper reports the first classroom deployment with 8 real students and the first cross-course scalability test across three courses spanning two academic levels and disciplines.
The evaluation uses RAGAS metrics on 660 questions. Answer Relevancy and Context Precision remained stable (0.88–0.94 and 0.88–0.90 respectively). However, Faithfulness declined from 0.69 to 0.50 as curriculum distance from the original course increased. The authors suggest this reflects generation logic tuned to the original subject rather than a fundamental scalability flaw. The orchestration layer requires no modification, but achieving full course-agnosticism for downstream components needs further investigation. The paper is under review at SN Computer Science.
- First real-world classroom test of LEA with 8 students across 3 courses
- RAGAS evaluation of 660 questions: Answer Relevancy 0.88-0.94, Context Precision 0.88-0.90, Faithfulness drops from 0.69 to 0.50 with curriculum distance
- Orchestration layer is course-agnostic, but downstream generation components need tuning for cross-course reliability
Why It Matters
Real-world deployment exposes AI tutor accuracy gaps that simulations miss, critical for scaling adaptive learning systems.