FRESCO: Benchmarking and Optimizing Re-rankers for Evolving Semantic Conflict in Retrieval-Augmented Generation

Researchers from Meta Superintelligence Labs and UCLA have introduced FRESCO (Factual Recency and Evolving Semantic COnflict), a new benchmark designed to evaluate re-rankers in Retrieval-Augmented Generation (RAG) systems. Unlike existing benchmarks that test re-rankers in static environments, FRESCO specifically assesses performance in temporally dynamic contexts where information evolves over time. The benchmark pairs recency-seeking queries with historical Wikipedia revisions to test whether re-rankers can prioritize factually recent evidence while maintaining semantic relevance.

The study reveals a critical failure mode across existing re-rankers: they consistently demonstrate a strong bias toward older, semantically rich documents even when those documents contain factually obsolete information. This bias persists despite newer documents containing more accurate, up-to-date information. The researchers found this problem affects multiple re-ranker models currently used in production RAG systems.

To address this limitation, the team investigated an instruction optimization framework that identifies Pareto-optimal instructions balancing evolving and non-evolving knowledge tasks. Their approach achieved gains of up to 27% on evolving knowledge tasks while maintaining competitive performance on non-evolving knowledge tasks. This represents a significant advancement in making RAG systems more reliable for real-world applications where information constantly changes.