LFRAG boosts multimodal document RAG with block-level retrieval, 73% less tokens
What if the key to more efficient document retrieval wasn’t better language models, but smarter segmentation of the page itself? A new framework called LFRAG shows that moving from whole-page to block-level retrieval can simultaneously halve token usage and boost answer accuracy.
Existing multimodal RAG systems typically retrieve entire pages from visually rich documents, missing fine-grained semantic and layout structures. This leads to irrelevant context, wasted tokens, and reduced accuracy. To solve this, researchers from East China Normal University and Ant Group introduce LFRAG (Layout-oriented Fine-grained Retrieval-Augmented Generation). The framework first performs layout segmentation to break documents into semantically coherent blocks (e.g., paragraphs, tables, figures). It then uses a semantic-layout fusion encoder with cross-attention to combine local block semantics with global document context. Finally, block-level late interaction retrieval allows precise query-to-content alignment, cutting out irrelevant information before generation.
To evaluate LFRAG, the team created LFDocQA, a large-scale benchmark with block-level annotations across diverse document types like reports, invoices, and academic papers. On this benchmark, LFRAG achieved state-of-the-art retrieval performance, outperforming the best baseline by 7.20% in answer accuracy while reducing token consumption by 73.07% during generation. These results demonstrate that moving from page-level to block-level retrieval is both more accurate and significantly more efficient for multimodal document understanding. The code and dataset will be released soon, offering a practical upgrade for enterprise RAG systems handling complex layouts.
- Block-level retrieval reduces token consumption by 73% compared to page-level approaches, directly lowering API costs for document-heavy applications.
- LFRAG achieves a 7.20% improvement in answer accuracy on the LFDocQA benchmark, showing that finer granularity does not sacrifice relevance.
- The reliance on layout segmentation introduces potential failure modes for irregular documents and cross-block queries, requiring careful validation before wide adoption.
Why It Matters
LFRAG redefines RAG efficiency by prioritizing structural granularity over brute-force token usage, potentially reshaping document AI economics.