RAG debugging reveals chunking, vector search, stale index pitfalls

A developer on Reddit shared their painful journey debugging a RAG (retrieval-augmented generation) system, revealing three common but poorly documented pitfalls. First, fixed-size chunking splits text based on token count, not where a thought ends. This often cuts the sentence containing the answer into the next chunk, which doesn’t make the retrieval cutoff. The model then receives half the context and hallucinates the rest. The developer spent weeks assuming it was an embedding problem before actually inspecting the retrieved chunks and finding the answer split across chunks.

Second, vector search alone fails for exact identifiers such as version numbers or product codes. Semantic search returns “close” matches, but “close” is wrong for precise lookups. The fix? Hybrid search: combining BM25 keyword matching with vector embeddings. Third, a stale index caused two days of confidently wrong answers after updating a document without re-indexing. The developer emphasizes that these issues are not hard to solve, but are rarely mentioned in introductory RAG tutorials.