The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness

Current AI tutor evaluations focus almost exclusively on the pedagogical quality of feedback messages—how accurate, clear, or helpful the advice seems in isolation. But as a new study from researchers at UC Berkeley, Aalto University, and others argues, this misses a crucial dimension: what do students actually do with the feedback? The team analyzed 10,235 student code submissions from an introductory programming course, each paired with AI tutor feedback, to measure behavioral signals like whether students attempted to fix errors and whether those fixes were applied correctly.

Comparing two different AI tutors deployed across separate semesters, the researchers found substantial differences in student engagement that no pedagogy-only evaluation would have caught. More importantly, these behavioral signals—action and correctness—were more strongly correlated with students' perception of helpful feedback than any pedagogical metric alone. The work, accepted to the 27th International Conference on Artificial Intelligence in Education (AIED 2026), argues for adding a "behavioral axis" to tutor evaluation. This could help educators and developers identify which tutors actually drive learning improvements, not just which ones sound good on paper.