New study: GPT-4.1 lags ConvBERT in predicting item difficulty

Estimating item difficulty is critical for both classroom formative assessments and high-stakes summative tests. In a new arXiv study, Xinyi Wang and colleagues examined whether large language models can accurately predict difficulty levels using items from a large-scale Reading and Writing test. They tested multiple prompting strategies and parameter settings across several LLMs, comparing their performance to encoder-only models like ConvBERT and feature-based supervised machine learning baselines. The best LLM result came from zero-shot GPT-4.1 with temperature set to 0, achieving a quadratic weighted kappa (QWK) of 0.578. However, that fell short of ConvBERT's 0.625, which also beat the strongest feature-based model. This indicates that semantic understanding alone, as captured by generative LLMs, is currently inferior to specialized discriminative encoders for this task.

Further analysis revealed systematic weaknesses. All LLMs mislabeled difficult items, and notably GPT-5.4, despite its advanced capabilities, tended to underestimate item difficulty—essentially treating most items as easy. Dimensionality reduction of item embeddings showed that representations from different difficulty levels were thoroughly mixed, suggesting that semantic text content alone carries insufficient signal to infer difficulty. The authors conclude that caution is warranted when using LLMs to automatically generate test items with targeted difficulty levels, particularly as model capabilities scale. While LLMs may write fluent questions, they cannot reliably judge or control the difficulty of what they produce, which is a critical limitation for automated item generation pipelines in education.