Assessment Design in the AI Era: A Method for Identifying Items Functioning Differentially for Humans and Chatbots

A team of researchers has published a groundbreaking paper titled 'Assessment Design in the AI Era: A Method for Identifying Items Functioning Differentially for Humans and Chatbots.' The study applies a core concept from psychometrics—Differential Item Functioning (DIF)—traditionally used to detect test bias across human demographic groups, to the new challenge of distinguishing AI from human performance. By analyzing responses from six top chatbots (ChatGPT-4o & 5.2, Gemini 1.5 & 3 Pro, Claude 3.5 & 4.5 Sonnet) on a high school chemistry diagnostic and a university entrance exam, the method statistically identifies test items where AI and human responses systematically diverge.

This DIF analysis, combined with expert review, reveals the specific task dimensions that make questions particularly easy or difficult for generative AI. For instance, it can flag questions that rely on rote memorization or pattern recognition where AI excels, versus those requiring complex, multi-step reasoning or genuine conceptual understanding where it may falter. The result is a robust, theory-grounded framework that moves beyond simple benchmark scores. It provides assessment designers with actionable analytics to understand where their tests are most vulnerable to AI misuse and to redesign items for greater validity and fairness in an era where AI tools are ubiquitous.