Gemini-2.5-pro powers AI tutor training assessment, boosting real-world skills 7.4%

Researchers at Carnegie Mellon University developed an AI-driven assessment system that uses Google's Gemini-2.5-pro to evaluate human tutors' performance during training and in real-life tutoring sessions. Unlike traditional platforms that only assess online quizzes or simulations, this system analyzes transcriptions of authentic remote math tutoring. In a study with 86 tutors completing six scenario-based lessons, the AI measured skill transfer, finding a significant 7.4% learning gain from training. Using mixed-effects models across 405 session-to-lesson pairs, training performance predicted real-life transcript scores with an effect size of 0.25 SD. The best predictive model averaged open-response and multiple-choice performance, with open responses being more predictive.

After training, tutors were significantly more likely to encounter pedagogical opportunities (61.1% to 68.9%) and demonstrated higher execution quality (65.5% to 68.1%). Interrupted time series analysis suggested improvements were gradual rather than immediate post-training. The system leverages Gemini-2.5-pro to analyze nuanced open responses and authentic dialogues, providing a scalable way to link training to real-world application. The researchers contributed open datasets, AI prompts, and scoring rubrics to support transparency and reproducibility, marking a step toward data-driven tutor development.