New study: Evaluating AI solutions matches problem-solving in algorithms class
N=220 crossover study finds no learning loss when students critique GenAI outputs
A new arXiv paper (2607.27586) from Ethan Dickey, Marios Mertzanidis, and Alexandros Psomas asks whether teaching students to evaluate GenAI-generated solutions is as effective as having them solve problems themselves. The researchers ran a randomized A/B crossover study with 220 students in a junior-level algorithms course. Working groups either solved challenging algorithmic problems directly or evaluated often-flawed GenAI solutions, with roles swapped midway through the semester across six assignments.
Results showed no statistically significant differences between groups on midterm scores, final exam scores, overall course grades, or exam problems specifically aligned with the homework interventions. Students did score significantly higher on homework when evaluating GenAI solutions, but this localized advantage didn't translate into summative gains. Survey data revealed most students reported no change in study habits, though those who adapted their strategies rated the GenAI-evaluation assignments as significantly more helpful. The authors conclude that GenAI evaluation can be incorporated without broad performance losses, but meaningful learning gains require deliberate scaffolding beyond simple error diagnosis.
- Randomized A/B crossover study with N=220 in a junior-level algorithms course
- No significant differences in midterms, finals, or overall grades between solving and evaluating groups
- Higher homework scores from evaluation (but no downstream exam gains); adaptive students found it more helpful
Why It Matters
Shows AI-critique pedagogy is safe for coursework, but requires intentional scaffolding to boost genuine learning.