LLM judges biased by self/other labels, even without model names

A new arXiv preprint titled “Self- and Other-Labels Induce Bidirectional Bias in LLM Judges” (2608.18091), from Songeun Chae and four colleagues, tackles a stubborn problem in AI evaluation: self-preference. LLM-as-a-judge systems often rank their own outputs higher, but prior studies conflated that bias with style and quality differences in generated text. The authors bypassed this by having 10 LLMs judge narrative constraint selections—structured choices that carry no model-specific stylistic fingerprint but retain a recoverable, model-specific signature. This design lets them separate genuine self-preference from confounds like evaluator severity and selection quality.

Results were striking. Under blind evaluation, self-preference largely disappeared once selection quality and evaluator severity were controlled; it vanished on three of four rubric dimensions and even reversed on the fourth, where judges rated their own selections as less original. But when matched-quality selections were labeled as “self” or “other”—without naming any model—scores shifted bidirectionally. LLM judges inflated scores for self-labeled selections and deflated them for other-labeled ones, regardless of which model actually produced the selection. The authors conclude that authorship attribution itself is a distinct driver of evaluation bias, and that ground-truth-free, open-ended tasks can serve as controlled instruments for studying LLM judge behavior. This raises red flags for benchmark pipelines relying on LLM judges, where simple label cues—or even perceived ownership—could silently skew results.