New study warns: Visual RAG aggregation hides critical details in financial docs

Visual RAG, which treats entire documents as images and uses vision encoders to extract patch tokens, typically requires aggregating hundreds of patch tokens per document into a single vector for efficient storage and retrieval. A new paper by Ho Hung Lim and Yi Yang, accepted to Findings of ACL 2026, empirically investigates whether this aggregation sacrifices key information—particularly in financial documents where a single-digit change can flip meaning.

The authors created a diagnostic benchmark using financial documents with carefully controlled semantic shifts (e.g., changing ‘$10M’ to ‘$1M’). Their experiments reveal that single-vector aggregation collapses documents with different meanings into near-identical vectors, while patch-level retrieval reliably detects the changes. The root cause is identified as ‘global texture dominance’: the aggregation process averages out subtle local details, amplifying overarching visual patterns at the expense of fine-grained semantic signals.

These findings hold consistently across different vision model scales, retrieval-optimized embeddings, and multiple attempted mitigation strategies (e.g., weighted pooling, CLIP-based aggregation). The researchers conclude that single-vector visual document retrieval poses significant risks for financial applications, where precision matters. The work raises a critical design question for builders of multimodal RAG pipelines: when is aggregation acceptable, and when must you retain patch-level granularity?