Qwen3.5 LLM estimates test item difficulty by mimicking student error patterns

A new paper from Christopher Ormerod and YoungKoung Kim demonstrates that a fine-tuned multimodal LLM based on Qwen3.5 can reconstruct classic item response theory (IRT) curves without explicit statistical fitting. The model is trained on a large corpus of multiple-choice questions containing both image and text stimuli, conditioned on labeled student ability levels. Instead of directly predicting correct answers, the LLM learns to simulate the systematic errors students make across a discrete range of ability levels. This lets the model implicitly capture the response probability curves defined by the three-parameter logistic (3PL) model and the multiple-choice model (MCM).

On a held-out test set, the model accurately approximates item difficulty from its predicted option probabilities alone, effectively replacing traditional parameter estimation software with a single neural pass. The approach leverages the LLM's existing semantic and multimodal understanding, but it doesn't just classify items—it aligns its outputs with the latent probability structures used in psychometrics. Accepted for AIME-Con 2026, the method suggests that large language models can serve as calibrated measurement tools. For assessment platforms, this could mean faster item pretesting, more efficient adaptive test construction, and a unified way to handle text-plus-image questions that standard IRT models struggle with. The authors note that future work could extend the method to other IRT variants and larger ability grids, potentially making LLM-driven psychometrics a practical alternative to dedicated estimation pipelines.