AuthorityBench: Benchmarking LLM Authority Perception for Reliable Retrieval-Augmented Generation

A team of researchers has introduced AuthorityBench, a comprehensive new benchmark designed to test a critical but often overlooked capability in Large Language Models (LLMs): their ability to perceive the authority of information sources. This goes beyond simple semantic understanding and is crucial for Retrieval-Augmented Generation (RAG) systems, which can be misled by low-quality or misleading external data. The benchmark comprises three distinct datasets: DomainAuth (10K web domains ranked by PageRank), EntityAuth (22K entities ranked by popularity), and RAGAuth (120 queries with documents of varying authority for real-world evaluation).

The researchers evaluated five leading LLMs using three different judgment methods: PointJudge, PairJudge, and ListJudge. The results showed that PairJudge and ListJudge methods, when combined with a PointScore output format, achieved the strongest correlation with ground-truth authority rankings. Notably, the study found that incorporating the actual text content of webpages consistently degraded an LLM's judgment performance, suggesting that authority perception is a distinct skill separate from analyzing writing style.

In practical downstream tests on RAG systems, the research demonstrated that filtering retrieved documents based on the LLM's authority perception led to substantial improvements in answer accuracy. This validates the core hypothesis: teaching models to weigh source credibility is a direct path to more reliable and trustworthy AI knowledge retrieval, moving beyond the current paradigm where any retrieved text is treated with equal weight.