RAG-DIVE: A Dynamic Approach for Multi-Turn Dialogue Evaluation in Retrieval-Augmented Generation

A research team including Lorenz Brehme, Benedikt Dornauer, and four others has introduced RAG-DIVE, a novel framework designed to solve a critical problem in AI evaluation: static datasets fail to capture how Retrieval-Augmented Generation (RAG) systems perform in real, back-and-forth conversations. Current methods rely on predefined, one-directional queries, which don't reflect the adaptive, context-dependent nature of tools like customer support bots or research assistants. RAG-DIVE addresses this by dynamically simulating user interactions.

The framework operates in two main stages. First, a 'Conversation Generator' LLM acts as a simulated user to create multi-turn dialogue, while a 'Validator' filters out incoherent outputs. Second, an 'Evaluator' assesses the RAG system's performance across the entire conversation, generating both per-turn and aggregated multi-turn metrics. This allows developers to see how well their system maintains context and accuracy over a series of related questions.

In validation experiments, the team demonstrated RAG-DIVE's practical utility. It successfully detected performance degradations caused by intentional system modifications (proven via an ablation study) and showed consistent evaluation results across repeated trials. When tested against a traditional static evaluation method on an industrial RAG system, both approaches revealed similar performance trends, confirming RAG-DIVE's reliability while offering a far more realistic testing environment.