HIERA's hierarchical multi-agent system boosts relevance judgment by 38%

A new arXiv paper titled "HIERA: Hierarchical Multi-Agent Relevance Assessment for Content Discovery Systems" introduces a structured approach to automated relevance judgment—a core task for search quality evaluation. Authored by Pritom Saha Akash and colleagues, HIERA addresses limitations of current LLM-based assessors that rely on flat aggregation, such as single-step prompting or voting ensembles. The framework deploys four specialized agents: a Relevance Judge that decides when deeper analysis is needed, a Query Analyzer, an Item Analyzer, and a Relation Analyzer that coordinates query and item insights with external knowledge before reaching a final relevance verdict. This hierarchical coordination is the key contribution, as ablation studies confirm that using the same agents without hierarchy degrades performance, proving the structure itself drives gains.

Evaluated across five public datasets (EVS, MSRD, ESCI, WANDS, Home Depot), HIERA consistently outperforms 11 baselines, with notable improvements of 10.2% on Home Depot, 4.8% on ESCI, and up to 38% on EVS (p < 0.05). More striking, hierarchical coordination yields a 12.7% improvement over uncoordinated multi-agent collaboration using identical agents and external knowledge. This suggests that agent orchestration—rather than raw model power—can meaningfully improve automated relevance assessment. For teams building content discovery systems, the paper offers a practical blueprint: instead of throwing more prompts or agents at the problem, a controlled hierarchy with a judge-relation loop can produce more accurate and consistent relevance scores, potentially reducing reliance on costly human annotation.