Error-Aware TF-IDF RAG slashes ASR errors by 18% with near-zero latency

End-to-end automatic speech recognition (ASR) systems often hallucinate rare entities and domain-specific terms, especially in low-resource languages. Existing retrieval-augmented generation (RAG) solutions either ignore phonetic misrecognitions with standard sparse retrieval or rely on heavyweight cross-modal embeddings that introduce high latency. Jafari-Raddani's work tackles both issues with a purely lexical, error-aware framework that explicitly targets phonetic and loop hallucinations. The core innovation is a symmetric text normalization module paired with a novel error-aware TF-IDF algorithm. By constructing a sparse diagonal penalty matrix from historical errors, the retriever mathematically gives higher weight to documents that correct specific high-risk misrecognition patterns—without needing dense embeddings.

Evaluated on the Persian subset of the FLEURS dataset, the method achieved dramatic improvements: the error-aware hit rate jumped from 53.7% to 90.9%, and the final word error rate dropped from 23.06% to 18.83% in end-to-end tests. The approach maintains near-zero inference latency because it avoids heavy cross-modal processing. This makes it a practical, low-cost enhancement for production ASR systems, particularly for languages where phonetic errors dominate. While the paper focuses on Persian, the technique is language-agnostic and can be extended to other domains where ASR hallucinates critical terminology.