ACL 2026 paper: AI moral reasoning evals miss context-sensitive norms

A new position paper from Aidan Kierans, Ritam Dutt, Kaley Rittichier, Shiri Dori-Hacohen, and Avijit Ghosh argues that current evaluations of AI moral reasoning are only measuring half the problem. Written for the ACL 2026 Workshop on Evaluating Evaluations (EvalEval), the study distinguishes between the moral value problem—whether LLM outputs align with human values—and the moral norm problem—whether models can identify and correctly apply context-sensitive moral norms. The authors contend that the field has leaned heavily on descriptive ethics frameworks like Moral Foundations Theory and Kohlberg's stages of moral development, which emphasize value representation over normative application.

To support this, they reviewed existing benchmarks and evaluation methods, finding they cluster overwhelmingly around the value problem. The paper identifies three critical gaps: the absence of high-quality ground-truth data for moral norms and their applications, insufficient evaluation of intermediate reasoning processes, and limited attention to identifying morally relevant features in context. As a remedy, they propose a research agenda including standardized formal representations for normative theories, expert-annotated datasets capturing norm application, and evaluation protocols that explicitly separate values-level from norms-level competence. The goal is to push the field toward more systematic study of normative reasoning in LLMs, which has real-world stakes as AI systems are deployed in healthcare, law, and public policy.