CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

A research team led by Jitian Zhao, Changho Shin, and colleagues has published a groundbreaking paper titled 'CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation' on arXiv. The work tackles a critical flaw in the current standard for scalable AI evaluation: the LLM-as-a-judge paradigm. The researchers demonstrate that standard aggregation methods like majority vote or averaging are fundamentally flawed because they assume LLM judges provide independent estimates. In reality, these judges exhibit correlated errors caused by shared latent confounders—such as verbosity preferences, stylistic biases, or common training artifacts—which causes systematic mistakes to be amplified rather than corrected.

The CARE framework explicitly models LLM judge scores as arising from both a latent true-quality signal and these shared confounding factors. It provides theoretical guarantees for identifiability and finite-sample recovery under shared confounders, quantifying the systematic bias incurred when aggregation models ignore these latent factors. Tested across 12 public benchmarks spanning continuous scoring, binary classification, and pairwise preference settings, CARE consistently improved aggregation accuracy, reducing error by up to 26.8%. The team has released the code publicly, offering a practical tool for researchers and developers to obtain more reliable, less biased evaluations of models like GPT-4, Claude 3, or Llama 3, which is crucial for benchmarking progress and deploying trustworthy AI systems.