Sokipriala Jonah study: Multimodal graph-RAG shows limited gains on document QA

A new arXiv preprint from Sokipriala Jonah (submitted July 18, 2026) systematically evaluates when multimodal and graph-augmented retrieval-augmented generation (RAG) actually helps document question answering. The researchers built an explainable multimodal graph-RAG architecture that augments a text-only baseline with LLM-extracted knowledge graph triples (subject-relation-object) and CLIP-based retrieval of figures and tables. All three evidence sources are retrieved independently and fused only at generation time, enabling separate evaluation of each component's contribution. The controlled four-way ablation study uses 1,000 PubLayNet pages across three question types (single-passage, multi-hop, and figure questions) with two closed-weight and two open-weight multimodal generators.

The results are sobering for teams investing heavily in knowledge graph augmentation: adding graph evidence provided no reliable accuracy improvement across any generator or question type on this corpus. Multimodal retrieval of figures and tables offered only marginal gains—on pixel-only figure questions (where captions are stripped to test genuine visual understanding), text-only systems achieve zero accuracy, while multimodal systems reach just 0.057–0.114. This is constrained by both image retrieval performance (Recall@3=0.371) and the generators' limited ability to interpret dense scientific figures and tables. The study also highlights a practical cost concern: processing the same image produces an elevenfold variation in input tokens across generators, meaning image tokenization can dominate deployment costs. Crucially, caption-derived questions substantially overestimate text-only visual QA capability, warning against flawed benchmark design.