CSTutorBench benchmark shows small models falter at deeper tutoring

A new benchmark called CSTutorBench, developed by researchers H. Chad Lane and Bryson Kageler, systematically evaluates small language models (SLMs) as AI tutors for block-based programming in VEX VR, a K-12 robotics environment. The benchmark comprises 17 scenario-based questions scored against a pedagogical rubric grounded in established tutoring and feedback research, using a human-in-the-loop LLM-as-judge pipeline. Preliminary results across 11 models ranging from 4B to 120B parameters reveal that while SLMs perform well on surface-level criteria like vocabulary and tone, they consistently struggle with deeper pedagogical behaviors—particularly avoiding answer leakage and engaging with student debugging histories.

Interestingly, the study found that model family and instruction-tuning approach were better predictors of tutoring quality than raw parameter count, though the small sample size limits the strength of that conclusion. A targeted prompt revision based on recent educational prompt engineering research improved scores for 10 out of 11 models, showing the importance of context-aware design. These findings underscore the value of pedagogically grounded benchmarks for selecting SLMs in real educational deployments, especially when privacy and cost make large proprietary models impractical.