Ceci n'est pas une explication: Evaluating Explanation Failures as Explainability Pitfalls in Language Learning Systems

A new academic paper from researchers Ben Knight, Wm. Matthew Kennedy, and James Edgell introduces L2-Bench, a comprehensive benchmark for evaluating AI-generated feedback in language learning systems. The benchmark assesses six critical dimensions: diagnostic accuracy (correctly identifying errors), awareness of appropriacy (understanding context-appropriate language), causes of error (explaining why mistakes occur), prioritization (focusing on important issues first), guidance for improvement (actionable next steps), and supporting self-regulation (helping learners monitor their own progress). The authors argue that AI systems often fail in these areas, producing explanations that appear helpful but are fundamentally flawed—what they call 'explainability pitfalls.' These failures can lead to attainment harms (slower learning), human-AI interaction harms (eroded trust), and socioaffective harms (frustration or demotivation).

Accepted to the MIRAGE Workshop at IUI 2026, the paper highlights how the specific context of language learning amplifies these risks, as learners may lack the expertise to detect subtle errors in AI feedback. The researchers call for more robust evaluation frameworks that go beyond surface-level accuracy to assess the real-world impact of AI explanations on learning outcomes. With millions of learners worldwide relying on AI-powered tools like Duolingo and ChatGPT for language practice, the study underscores the urgent need for developers to prioritize safety, trustworthiness, and effectiveness in explanatory AI systems. The L2-Bench benchmark provides a structured approach to identifying and mitigating these pitfalls before they cause widespread harm.