LLM-as-a-Judge Exposed: Formalism Trap Blinds Eval Systems Under Social Load

A new paper from researchers Dahlia Shehata and Ming Li (arXiv:2607.28641) exposes a critical flaw in LLM-as-a-Judge evaluation systems: the Agentic Formalism Trap. The authors argue that when these systems are subjected to social load—essentially pressure from consensus or procedural conventions—they conflate structural proceduralism with actual semantic correctness. To quantify this, they introduce the Evaluative Dissonance Index (D_E), which measures the gap between what an evaluator appears to validate and what is objectively true. Their analysis spans 22,500 trajectories across three benchmark domains: GAIA, SWE-bench, and Multi-Challenge, they used deterministic lexical grounding to validate a taxonomy of hallucination maneuvers with overwhelming statistical significance (p < 10^-120).

Using a logistic meta-evaluator, they isolated the exact syntactic triggers that cause evaluator capture, achieving a ROC-AUC of 0.8779. Critically, a zero-shot Leave-One-Domain-Out transfer experiment showed the vulnerability is universal—not tied to a single benchmark—with a mean ROC-AUC of 0.7482 across unseen domains. Architectural profiling further revealed that different simulated swarm topologies (how multiple LLM judges interact) produce mathematically distinct semantic blind spots. This proves that unanchored closed-loop evaluation is not just fragile but systemically divergent, and it requires architecture-specific vigilance filters to prevent evaluator collapse. The findings have major implications for any AI pipeline that relies on LLM judges for benchmarking, reinforcement learning feedback, or safety evaluations.