arXiv study: LLMs match human moral labels but diverge on reasoning
LLMs often agree with humans on right versus wrong—but for completely different reasons.
A new paper, "Agreement Is Not Alignment: Divergent Moral Grounds in Human and LLM Ethical Judgments," challenges the common practice of using agreement with human judgments as a proxy for AI alignment. The researchers—Octavian M. Machidon, Alina L. Machidon, Vojko Strahovnik, Mateja Centa Strahovnik, Jonas Miklavčič, and Marko Robnik Šikonja—built a curated 500-item benchmark derived from ETHICS, spanning five domains of moral judgment. They collected fresh annotations from both humans and LLMs, capturing not just final labels (e.g., "acceptable" or "unacceptable") but also the rationales behind those judgments.
Across frontier and open model families, LLMs often agreed with the human annotator majority on final labels. However, rationale-level analysis exposed a systematic divergence in the moral grounds models invoked. Even when models reached the same verdict as humans, they redistributed attention across categories such as harm, respect, promise-keeping, justice, desert, and excuse relevance. This means two agents can reach an identical conclusion while appealing to fundamentally different principles or contextual assumptions. The authors argue that label-based evaluation can therefore be misleadingly reassuring, and that true alignment requires comparing the reasons, principles, and moral priorities expressed in model judgments—not just the outcomes. The paper was accepted and presented at the AI Transparency Conference (AITC 2026) in Nuremberg, Germany, and is available on arXiv with the identifier 2608.12368.
- 500-item ETHICS-derived benchmark spans five moral judgment domains with fresh human and LLM annotations of labels plus rationales
- Models frequently matched human majority labels but systematically diverged on moral grounds like harm, justice, and excuse relevance
- Authors conclude label-based evaluation is misleadingly reassuring and call for rationale-level analysis to assess true alignment
Why It Matters
Accuracy on moral labels can hide misaligned reasoning, so AI safety evaluations must inspect underlying justifications, not just outcomes.