New training technique stops AI language graders from using shortcuts

Automated speech and language processing systems increasingly use raw audio or text as input, often powered by complex transformer-based models. While highly effective at learning non-linear mappings, these models can also pick up 'shortcuts'—overly relying on spurious input features that correlate with the target score rather than actual proficiency. For language proficiency assessment, this creates a vulnerability: learners can artificially boost scores by exploiting these shortcuts without improving their language skills.

Gao, Gales, and Knill from Cambridge propose a novel training criterion that explicitly penalizes the model's reliance on such shortcut features. They tested it on two assessment systems: one using audio input and one using automatic speech recognition (ASR) text. Both systems initially showed correlation with exploitable features far above the human reference. After applying the modified training criterion, the correlation dropped to near-human levels. This work offers a practical defense against AI grading manipulation, ensuring that automated assessments more accurately reflect genuine second-language proficiency.