CourseKata study: Text features boost quiz prediction by 9.1%

A new arXiv paper from Samin Khan demonstrates that educational performance prediction can be significantly improved by analyzing the content of assessment questions, not just students' past scores. Using 4,742 student-chapter observations from the CourseKata platform, the study extracted lightweight linguistic features from review-question wording and visual features from textbook images. When added to a baseline model that only used a student's average prior exercise performance, these content features improved quiz score prediction by 9.1% in a five-fold cross-validation setup. The work underscores that assessment content itself—varying in linguistic complexity and visual detail—carries predictive signal beyond historical behavior.

The findings reveal an important nuance: text features consistently reduced prediction error in leave-chapter-out validation, while models incorporating image features actually performed worse than the text-only baseline. This suggests that textbook visuals may add noise rather than signal for performance prediction, possibly because complex diagrams or illustrations confuse rather than clarify when tied to quiz questions. For educational technologists, the results advocate for lightweight, context-aware models that integrate question difficulty, wording style, and readability metrics. The paper (4 pages, 2 figures) is a practical step toward more intelligent tutoring systems that adapt to both student history and the content they're being tested on.