LLM judges fabricate rationales: Study finds cue-invariance failure, proposes fix

A new arXiv paper by Tapwal, Kumar, and Maple asks a critical question: are LLM judges faithful to the text or do they fabricate rationales based on spurious cues? The authors introduce a causal framework with five novel cue interventions—Blind, Truth, Flip, Placebo, and Reveal-After—to probe whether judge rankings and explanations remain stable when non-evidential cues are perturbed. Testing on a dataset of 1,000 summaries from both extractive models and LLMs, they find substantial cue-anchored rationalization: judges adjust rankings based on verbosity and confidence cues rather than true content. The study also introduces tie-aware metrics like label-aligned rhetoric and explanation drift to quantify both outcome anchoring and rationale anchoring.

To combat this, the team proposes a structured mitigation called PROOF-BEFORE-PREFERENCE, which enforces an evidence-lock phase before scoring and ranking. Compared to standard chain-of-thought prompting, this method markedly improves cue invariance, reducing rationalization bias by anchoring decisions to actual text evidence. The findings have major implications for automated evaluation pipelines in summarization and dialogue, where LLM judges are increasingly relied upon for benchmarks. The paper suggests that without such safeguards, judges may systematically prefer style over substance, undermining the validity of evaluation results.