Multimodal CoLRAG-TF: Triple-Filtered RAG Boosts PDF Search by 71.6%

Multimodal CoLRAG-TF tackles the long-standing challenge of retrieval-augmented generation over complex, heterogeneous PDF collections. Unlike traditional RAG systems that rely solely on text embeddings or keyword matching, this new architecture from researcher Takato Yasuno integrates four complementary retrieval axes: dense text embeddings (via FAISS), BM25 keyword matching, OpenIE knowledge-graph triple filtering (11,414 triples indexed for sub-second lookup), and image-based similarity from a hybrid OCR pipeline with LLM-generated captions. A HippoRAG2-inspired coarse-to-fine retriever narrows the search space from volume to chapter to block before final fusion scoring. Bayesian optimization of fusion weights revealed that the triple axis must dominate (α=0.44) to counter lexical bias and sustain multi-hop retrieval quality.

Evaluated on a 457-pair benchmark derived from 43 Japanese disaster lesson PDFs, Multimodal CoLRAG-TF achieved a retrieval recall of 0.9909 and a 71.6% improvement in multi-hop answer similarity over single-hop queries. An image-to-lesson pipeline using a vision LLM further demonstrated the system's applicability to visual inputs. These results establish triple-filtered multimodal fusion as a critical technique for structured reasoning over noisy, domain-specific documents. While the current implementation focuses on disaster preparedness materials, the framework is designed to generalize to other complex document domains such as legal, medical, or technical documentation.