LayerRAG-Bench exposes why agentic RAG fails beyond schema drift

Musa Shams' new arXiv paper, LayerRAG-Bench, delivers a systematic stress test for agentic retrieval-augmented generation systems. Agentic RAG systems combine LLMs with external tools, databases, and memory, and they can fail in ways that plain RAG doesn't. The benchmark isolates five reliability layers: evidence, tool-contract, authorization, session-state, and schema. It spans 8 enterprise domains, 240 tasks, 9 fault scenarios, and 2 contract modes, producing 38,880 live task-level records across nine models from OpenAI, Anthropic, and Gemini. The result is the most granular public measurement yet of how these agents break in production-like conditions.

The most striking finding: schema normalization alone lifts schema-drift success from 0.000 to 0.913, a massive improvement. But it does nothing for stale evidence, missing tool output, denied permissions, or wrong-session context. These failures persist even when the model's answers look grounded — meaning evaluation methods that only check groundedness produce substantial false positives. The paper concludes with a layer-specific evaluation principle: each reliability intervention should be credited only for fixing its target layer, not advertised as a universal fix. For engineers building agentic RAG pipelines, this benchmark offers a concrete checklist for where to invest in reliability tooling and how to avoid overestimating model performance from surface-level metrics.