New method cuts VLM costs by 70% while matching accuracy in multimodal RAG
Relevance ≠ utility — researchers fix costly image overuse in RAG systems.
Deep Dive
Xue Li and Yiming Gai propose post-hoc selective modality escalation for multimodal RAG. Instead of routing based on pre-retrieval image relevance (which over-escalates substantially relative to an oracle), they first answer cheaply from text/tables, then verify if an image is needed. On the MultiModalQA benchmark, their router matches the accuracy of an always-on VLM pipeline but issues far fewer visual calls, closing most of the gap to an oracle escalation rate.
Key Points
- Pre-retrieval image relevance over-escalates by 40%+ compared to an oracle decision rule.
- Post-hoc method recovers always-on VLM accuracy while issuing 70% fewer visual calls on MultiModalQA.
- Verifier checks (query, draft answer, evidence) to localize missing modality before escalating.
Why It Matters
Cheaper multimodal RAG without sacrificing accuracy — critical for cost-sensitive enterprise AI pipelines.