GraphRAG vs RAG: New study shows wins, but LLM judges are wildly unstable
A 5,000+ judging study finds LLM citation scores flip 41% of the time — even for the same model
A new arXiv paper from Meftun Akarsu and colleagues (arXiv:2608.00705) takes a hard look at the contradictory results reported for GraphRAG versus vector RAG. The team ran a triple-robustness analysis that varied embedders (local e5-small vs. Azure text-embedding-3-small), corpora (DO-178C requirements vs. MuSiQue Wikipedia chains), and judges (paired GPT-5.4 and GPT-4.1), spanning 2x4,440 main-matrix runs and over 5,000 faithfulness judgments.
The headline finding: GraphRAG's graph walk floods the context window with low-precision retrieval (0.12-0.23), but its synthesizer cites selectively (0.48-0.65). That means simply scoring the retrieved set as the attribution set flips the architecture ranking — reconciling some earlier disputes. Answer-level results were corpus- and strategy-conditional but embedder-robust: GraphRAG tied vector RAG on short-hop DO-178C queries but won every MuSiQue stratum. Agentic pipelines only led on 3+-hop requirements. Even more troubling, single-judge LLM faithfulness proved fragile: GPT-5.4's self-kappa across embedders was 0.137 (41% verdict change) against a same-day test-retest floor of 0.76, and re-judging frozen inputs 11 weeks later yielded kappa ≤0.14 for both judges. The paper argues RAG claims need this level of robustness testing — including the citation-measurement point — before they're trusted.
- GraphRAG's graph walk retrieves at precision 0.12-0.23, but its synthesizer cites selectively at 0.48-0.65 — inverting rankings if you measure the retrieved set alone.
- Across 4,440 runs, GraphRAG tied vector RAG on short-hop DO-178C queries and won every MuSiQue stratum; agentic pipelines led only on 3+-hop requirements.
- Single-judge LLM faithfulness is fragile: GPT-5.4's self-kappa was 0.137 (41% verdict change) vs. a 0.76 test-retest floor, and both judges dropped to kappa ≤0.14 after 11 weeks.
- A learned router on dense embeddings alone reached macro-F1 0.86 for hop classification.
Why It Matters
RAG benchmark claims without multi-judge, multi-corpus validation may mislead AI teams — a caution for production choices.