New IBA Method Boosts Knowledge-Based VQA Without Training
Training-free entity identification beats fine-tuned multi-modal re-ranking on two benchmarks.
Knowledge-Based Visual Question Answering (KB-VQA) requires grounding visual queries to external knowledge beyond what's directly visible in images. While multimodal large language models (MLLMs) have strong perceptual abilities, they struggle with fine-grained entity and evidence grounding. Existing multimodal retrieval-augmented generation (MM-RAG) methods tightly couple entity discrimination and evidence ranking into a single re-ranking stage, leading to high costs and limited generalization.
Researchers from the paper "Ground Then Rank" (accepted at ACL 2026 Findings) propose the Identify-Before-Answer (IBA) framework. IBA decouples entity identification from section-level re-ranking: an MLLM selects high-confidence entities from candidate names, then an off-the-shelf textual re-ranker handles evidence selection. This simple, training-free approach consistently outperforms fine-tuned multi-modal re-ranking baselines on Encyclopedic-VQA and InfoSeek, while reducing both training and inference complexity. The authors also release their implementation for reproducibility.
- IBA framework decouples entity identification from evidence ranking, reducing complexity vs. traditional MM-RAG.
- Outperforms fine-tuned baselines on Encyclopedic-VQA and InfoSeek while being training-free.
- Improvements come from better entity identification and more informative evidence selection once entity is fixed.
Why It Matters
Enables more efficient and accurate knowledge-based visual QA without costly fine-tuning or complex pipelines.