Enhancing Scientific Literature Chatbots with Retrieval-Augmented Generation: A Performance Evaluation of Vector and Graph-Based Systems

A new research paper from Hamideh Ghanadian, Amin Kamali, and Mohammad Hossein Tekieh presents a comprehensive evaluation of retrieval-augmented generation (RAG) systems for scientific literature chatbots. The study, published on arXiv (ID: 2602.17856), systematically compares vector-based and graph-based retrieval approaches, proposing a hybrid system that leverages both structured (graph) and unstructured (vector) databases to access scientific articles and gray literature.

The researchers created benchmark test sets using a GPT model, with outputs annotated for evaluation. They examined two distinct use-case scenarios: retrieval from a single uploaded document and retrieval from large-scale corpora. The comparative analysis focused on retrieval accuracy and response relevance metrics, providing concrete insights into the strengths and limitations of each architectural approach. The hybrid system demonstrated superior performance in efficiently triaging sources according to specific research objectives.

This research matters because scientific literature continues to grow exponentially, creating accessibility challenges for researchers and professionals. Current chatbots often struggle with complex scientific queries that require understanding relationships between concepts and citations. The hybrid RAG approach addresses these limitations by combining the semantic search capabilities of vector databases with the relational understanding of graph databases. The findings suggest that such systems could significantly improve evidence-based decision making across scientific fields, potentially accelerating research discovery and knowledge synthesis.