LLM yes-no bias is a surface artifact, not a moral flaw

A new study from Haonan Huang on arXiv (2607.05552) tackles the well-documented yes-no bias in large language models. The paper introduces a psychometric battery called 'crossed symmetrization' that flips every logically irrelevant factor (order, wording) in balanced pairs across question forms. This allows the researchers to separate three confounded components: order bias (tendency to pick the last printed option), lexical pull (attraction to the word 'no'), and the actual logical verdict. Using graded ratings across logically equivalent forms, the study recovers a coherent internal moral scale θ for each model.

The results reveal that frontier models like GPT-5.5 and Gemini show nearly format-invariant moral stances (cross-form incoherence 0.12–0.21 on a ±1 axis). However, Claude models (Claude 3.5 Sonnet, Claude Opus 4) exhibit a substantial artifact (story-averaged −0.32 to −0.86), driven by a strong order bias toward the last-printed option—opposite to classic human primacy—plus a lexical pull toward the word 'no'. Crucially, when the words 'yes' and 'no' are swapped for arbitrary labels, the logical bias vanishes (≈0 for all frontier models), proving the pull is surface-level, not a rejection bias.

The study also finds that extended reasoning (chain-of-thought) shrinks this artifact, and introduces a minimal model P = σ((θ ± m)/s) that quantifies framing susceptibility m separately from moral decisiveness s. The authors conclude that measuring what an LLM values requires crossing the frames of the question, not asking once. This has direct implications for deploying LLMs in high-stakes moral decision-making.