AI tutors miss flawed reasoning when students get right answers, study finds

A new study published at AIED'26 by researchers Moiz Imran and Sahan Bulathwela reveals a critical blind spot in intelligent tutoring systems: the "correct answer trap" (CAT). When students arrive at a correct final answer through flawed reasoning, AI tutors consistently fail to flag the underlying misconception. Analyzing real student responses from the Eedi mathematics platform, the team found that 71% of these failures occur in just two question types—both share a structure where incorrect logic inadvertently yields the right numerical answer.<br><br>Comparing a fine-tuned T5 model with a frontier large language model, the study found that improved capabilities reduce but do not eliminate the problem. The frontier model achieved only 57% detection accuracy versus 84% for the fine-tuned T5, and even the best model generated roughly four false alarms for every genuine detection. The authors warn that high overall accuracy can mask these critical failures, and that automated reasoning assessment still requires human judgment for practical deployment at scale.