Study finds simple chunking matches advanced methods for RAG on academic texts
Cluster-based semantic chunking failed to beat fixed-size or recursive strategies in rigorous testing.
A team of researchers from the University of Applied Sciences Upper Austria—Valentin J.J. Kreileder, Johannes Reisinger, and Andreas Fischer—published a paper on arXiv evaluating chunking strategies for Retrieval-Augmented Generation (RAG) on long academic texts. RAG systems enhance LLMs by retrieving relevant document chunks before generating answers. The study tested three chunking methods: fixed-size, recursive (splitting at sentence boundaries), and cluster-based semantic chunking (grouping related content via embeddings). The benchmark used the RAGAs (Retrieval Augmented Generation Assessment) framework on a set of structured academic theses.
The key finding: cluster-based semantic chunking did not outperform the simpler fixed-size or recursive approaches. The RAGAs faithfulness metric showed limited reliability in this academic setup, raising questions about its broader applicability. Performance varied substantially between fixed questions and document-specific questions, likely due to formatting and preprocessing differences. The authors conclude that under their tested configurations, extra complexity of semantic chunking offers no advantage. This suggests practitioners evaluating RAG on similar content may achieve equal results with straightforward chunking strategies, especially when document structure is already clear.
- Cluster-based semantic chunking showed no performance gain over fixed-size or recursive chunking for RAG on academic theses.
- RAGAs faithfulness metric was unreliable in this academic text scenario.
- Performance differences between fixed and document-specific questions were substantial, likely tied to document formatting and preprocessing.
Why It Matters
For RAG practitioners, simpler chunking may work as well as costly semantic methods—save compute, not quality.