Detecting AI-Generated Essays in Writing Assessment: Responsible Use and Generalizability Across LLMs

A new research paper titled 'Detecting AI-Generated Essays in Writing Assessment: Responsible Use and Generalizability Across LLMs' tackles a critical problem in education and content verification. Authored by Jiangang Hao, the study provides an overview of current AI essay detectors and guidelines for their responsible use, then dives into empirical testing. The core finding is that detectors trained specifically on essays from one large language model (LLM), such as GPT-4, show poor performance when trying to identify essays generated by other state-of-the-art models like Anthropic's Claude 3.5 or Meta's Llama 3. This creates a major reliability gap for tools meant to ensure academic integrity.

The research methodology used public GRE writing prompts to generate essays across different LLMs, then evaluated detector generalization. The results indicate that achieving robust detection in practice requires continuous retraining and adaptation as new models emerge. For educators and assessment platforms, this means a static detector is insufficient; they need systems that can evolve. The paper's 21-page analysis offers concrete guidance for developing more practical, generalizable detection systems, highlighting that responsible use must account for this rapidly shifting technological landscape to avoid false accusations and maintain assessment validity.