RAG study finds GraphRAG over-cites and fails multi-hop tasks
New research reveals GraphRAG's 11-15 citations per answer vs 0.12-0.23 precision...
New research titled *Universal Pathologies, Conditional Consequences* by Meftun Akarsu and Burak Ozdemir systematically dismantles assumptions about GraphRAG’s superiority in multi-hop question answering. The authors conducted a triple-robustness analysis—varying embedders (local e5-small to Azure text-embedding-3-small), corpora (DO-178C typed-edge requirements to Wikipedia via MuSiQue), and judges (paired GPT-5.4 vs GPT-4.1)—across 4,440 main-matrix runs, 600 cross-corpus runs, and 1,200 faithfulness judgments.
Their findings reveal systemic flaws: GraphRAG consistently over-cites, emitting 11–15 document IDs per answer while achieving only 0.12–0.23 citation precision across all configurations. Faithfulness drops precipitously in domain-specific corpora like DO-178C (74%→40% across hops), though it paradoxically improves on Wikipedia chains (42%→58%) because over-cited paragraphs remain topically relevant. The study argues that triple-robustness testing is the minimum standard for evaluating trustworthy RAG systems.
- GraphRAG emits 11-15 citations per answer but achieves only 0.12-0.23 precision across 4,440 runs.
- Faithfulness collapses 74%→40% on DO-178C requirements due to structural over-citation.
- A learned router using dense embeddings alone achieves 0.86 macro-F1 on hop classification.
Why It Matters
GraphRAG's over-citation flaw threatens trust in AI citations—critical for regulated domains like aviation or healthcare where traceability is non-negotiable.