AI Judges Are Lying to Us: How AURA Exposes Their Hidden Biases

Large language models are increasingly used as judges for open-ended generation tasks, but their preferences remain imperfect proxies for human judgment. Existing auditing pipelines often assume that a reliable subset of examples or clean supervision signals are available beforehand, for example, from human annotation, heuristic filtering, or outputs of strong judges. In LLM evaluation, this assumption is fragile — the initial split may inherit judge bias, while human verification is typically too scarce to define stable groups at scale. To address this, researchers from multiple institutions propose AURA: an adaptive uncertainty-aware refinement framework for auditing pairwise LLM-as-a-judge decisions under selected human verification.

AURA iteratively learns a human-consistency signal, propagates reliable evidence, and prioritizes uncertain comparisons for human review. The core idea is to treat trust in a judge as a latent quantity that is progressively refined as evidence accumulates. The framework provides a compact formulation, a stable refinement procedure, and comprehensive evaluation on both synthetic and real pairwise LLM-answer data. Results show effective bias detection without requiring large-scale human annotation upfront. For practitioners, AURA offers a more scalable and robust way to audit automated evaluators, reducing the risk of relying on flawed LLM judgments.