New study reveals optimal text chunk size for RAG systems
Small chunks vs large chunks — which yields better RAG performance?
A new paper from researchers German Garrido-Lestache Belinchon and Hugo Garrido-Lestache Belinchon, published on arXiv, dives into a critical but often overlooked parameter in Retrieval-Augmented Generation (RAG) systems: text chunk size. RAG pipelines split source documents into segments before retrieval, and the granularity of those segments can dramatically affect both generation quality and computational efficiency. The study systematically evaluates how chunk size—ranging from individual sentences to entire chapters—impacts retrieval precision, contextual correctness, and overall output coherence.
The authors find a fundamental trade-off: smaller chunks narrow retrieval focus, boosting precision but limiting the model's ability to generate contextually rich responses. Larger chunks contain more comprehensive information, potentially improving factual correctness, but they also introduce extraneous noise and increase both retrieval cost and processing time. Additionally, the number of chunks returned to the LLM interacts with chunk size; adjusting both parameters together can yield better outcomes than optimizing either in isolation. The paper provides empirical evidence to guide practitioners in selecting chunk sizes that balance accuracy, coherence, and cost—making it a practical resource for anyone building or tuning RAG systems.
- Smaller chunks (e.g., sentences) increase retrieval precision but may degrade response coherence due to insufficient context.
- Larger chunks (e.g., chapters) offer broader context at the cost of added noise and higher computational overhead.
- The number of retrieved chunks interacts with chunk size—joint optimization is recommended for best RAG performance.
Why It Matters
For professionals building RAG pipelines, this study offers data-driven guidance to balance accuracy, coherence, and cost.