TR-RAG: Teacher-Regularized RL for Cross-Lingual RAG

Cross-lingual retrieval-augmented generation (RAG) systems often face a critical bottleneck: when users query in languages other than English but retrieved evidence is in English, models tend to drift into English or code-switched outputs and fail to reliably use evidence. A new paper from Haotian Zhou and colleagues identifies two core post-training challenges behind these failures: prefix-dependent errors that fixed-trajectory supervision cannot handle, and noisy credit assignment from sequence-level rewards. To solve this, they introduce TR-RAG (Teacher-Regularized Reinforcement Learning for RAG), which couples reward optimization with on-policy distillation. A compact student model samples on-policy answers, while a stronger frozen teacher provides a prefix-wise reverse-KL anchor. This setup is paired with a decomposed reward that blends language consistency, character 3-gram recall, and an LLM judge score for evidence-grounded correctness.

Tested on three benchmarks (BioASQ-ENKB5, Hotpot-ENKB5, and the naturally multilingual MKQA) with two different backbone models, TR-RAG significantly outperforms strong baselines. Crucially, the teacher anchor acts as a safety net: it prevents catastrophic collapses in language consistency (up to ~27 percentage points) that reward-only RL can suffer on in-domain languages, while on distant out-of-distribution languages, where reward-only RL stalls at the base model's ceiling, TR-RAG still improves evidence grounding. In a surprising result, the compact student model even surpasses its 70B teacher in character 3-gram recall. This work opens the door to more reliable multilingual AI systems without requiring massive models or parallel corpora.