New study reveals 'Cherry-pick Override' flaw in LLM judges under mixed evidence

A new paper by Haoran Xu on arXiv (2606.07834) uncovers a critical flaw in LLM judges called 'Cherry-pick Override' (CCO). When evaluating claims with mixed evidence (both supporting and refuting sources), these judges often incorrectly commit to a directional verdict—SUPPORTS or REFUTES—instead of returning the authorized non-directional verdict CONFLICTING. The study defines CCO under an explicit task contract and introduces a same-denominator diagnostic protocol with matched-coverage bootstrap. On AVeriTeC's Conflicting subset (150 claims), three-option judges return a directional verdict on more than 84% of mixed-evidence claims. Even under a typed schema, three-judge majority voting amplifies this directional commitment on AVeriTeC (0.887 vs. 0.840), though the result does not replicate on VitaminC-Mixed.

The paper walks through an intervention ladder of common single-channel fixes—typed vocabulary, panel aggregation, confidence thresholding, validator-only filtering—and finds each leaves a distinct residual failure. Panel aggregation suppresses single-judge CONFLICTING dissent in 48% of CCO cases, while confidence thresholding fails because the panel is well-calibrated for directional verdicts (ECE=0.07 on pure-S/R). A validator-as-classifier approach nearly halves pure-evidence accuracy. The proposed solution is a minimal two-channel reference probe that reaches operating points neither single channel can. Under a random-veto null, its promotion to CONFLICTING is structurally targeted on AVeriTeC (empirical p<1/2001). The paper argues for an external commitment-control layer that separates verdict generation from commitment authorization, using structural evidence and confidence as orthogonal channels with NO-COMMIT as a routed controller state.