RAG study reveals hidden failures beyond final answer accuracy

A new arXiv paper by Bharath Simha Reddy Muthyam conducts a systems-level audit of Retrieval-Augmented Generation (RAG) pipelines, moving beyond final answer accuracy to examine intermediate failures. The study runs 56 controlled experiments on a fixed 500-question QA dataset linked to 20,958 corpus contexts. It systematically varies chunk size, retrieval depth (top k), embedding-based reranking, probabilistic retrieval noise, and repeated seeded runs to measure impact on retrieval, context packing, and generation. The goal is to identify where and how RAG systems break under different configurations.

Key findings reveal troubling non-monotonic behavior: retrieval-oriented metrics improved with broader settings, but downstream exact-match and F1 scores often declined. Smaller chunk sizes caused preprocessing-induced answer loss, and retrieval corruption led to progressive degradation. Broader retrieval regimes also showed higher observed variance, making performance unpredictable. The paper argues that RAG evaluation must include sensitivity, robustness, stability, and multi-stage failure analysis rather than relying solely on final answer accuracy. This work provides concrete evidence that optimizing one stage can hurt another, a critical insight for engineers building production RAG systems.