New study uses response-time propensities to measure student effort in tutoring

Adaptive learning systems promise major gains but often fail because measuring genuine student effort is notoriously difficult. Traditional proxies like total time on task can't differentiate careful thinking from hitting a hard problem. In a new paper accepted at the ACM Learning @ Scale conference, researchers introduce response-time propensities—a step-to-step measure of how long a student typically takes relative to problem difficulty. Using logs from 8 classroom deployments of algebra tutoring systems across 6 U.S. schools (794 students from 2020–2023), they built hierarchical models to estimate these propensities at both student and knowledge-component levels. The approach adjusts for skill difficulty, isolating effort from problem complexity.

The results reveal a nuanced picture: response-time propensities are stable within students (moderate to strong correlations), validating them as an individual-differences metric beyond right/wrong answers. However, the relationship to learning isn't uniform. For high-proficiency students, slower propensities predict greater learning efficiency—consistent with constructive, deep processing. For low-proficiency students, slower speeds are weakly or negatively associated with learning, suggesting unproductive struggle or disengagement. Crucially, these effects are strongest early in practice sequences and fade later, highlighting an actionable window for intervention. The method provides a practical, log-based signal that adaptive systems can use to detect emerging disengagement and target support precisely when it matters most.