Text embeddings predict item difficulty with 53% accuracy, hitting 57% of reliability ceiling

A new evaluation framework from Shi-Ting Chen and Jinsong Chen tackles the cold-start problem in item calibration by predicting psychometric parameters directly from text embeddings. The approach combines regularized regression on item text embeddings, repeated cross-validated R² with resampling standard deviation, and two novel upper bounds: a reliability ceiling (derived from parameter standard errors) and a design ceiling (from simulation-based power calibration). Applied to a mathematics item bank (EEDI) and a medical licensure benchmark (BEA 2024), the framework shows item difficulty is highly predictable (R²=0.53, or about 57% of its reliability ceiling).

Crucially, the apparent hierarchy among parameters (difficulty > discrimination > pseudo-guessing) is revealed to stem from differences in target reliability rather than text signal strength: text uniformly recovers 57-63% of reliable variance across difficulty targets. The 3PL pseudo-guessing parameter has a reliability ceiling near zero, making it an unviable target with current precision. On BEA, embedding-based regression matches leaderboard RMSE despite explaining almost no variance, highlighting the need for scale-free metrics and explicit ceilings. The authors also demonstrate that a single train/test split can inflate R² by 0.1-0.15, underscoring the necessity of repeated cross-validation for calibration support applications.