Epi2Diff Framework Predicts Test Difficulty from AI Reasoning Traces

Predicting how difficult a test question will be for humans is a central challenge in educational assessment, but traditional methods rely on expensive human calibration or shallow text analysis. A new paper titled "Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction" proposes a novel solution: use the reasoning traces of Large Reasoning Models (LRMs) to model the cognitive burden a question imposes. The authors introduce Epi2Diff (Episode to Difficulty), a framework that breaks down an LRM's step-by-step reasoning into structured "cognitive episodes" — functional problem-solving states like planning, computation, or verification. These episodes capture reasoning scale, effort allocation, and state transitions, which are then combined with semantic item features to predict human difficulty ratings. The approach treats difficulty not as a static property of text but as an observable consequence of the problem-solving process.

In experiments on four real-world human difficulty datasets, Epi2Diff consistently outperforms strong baselines including fine-tuned small language models, LLM in-context learning, and supervised LLM adaptation. On SAT-derived classification benchmarks, it achieves an 8.1% average relative improvement over supervised fine-tuning. Crucially, the analysis reveals that harder items trigger more iterative, effortful, and "implementation-centered" episode dynamics — not merely longer responses. This insight provides interpretable evidence about why certain questions are challenging, moving beyond black-box predictions. The work, submitted to arXiv and led by Chenguang Wang, suggests that cognitive episodes in LRM reasoning traces offer a new, scalable lens for educational measurement, potentially reducing the need for costly human calibration in test design.