RIMS boosts small LLM RAG with smoother preference optimization

Small-scale language models (SLMs) offer efficiency for retrieval-augmented generation (RAG) in resource-constrained environments, but their limited capacity makes them vulnerable to noisy or spurious retrieved evidence. Existing preference-based methods like RoseRAG discard useful signal by selecting only the hardest single preference pair via hard argmin/argmax, or treat multiple pairs as independent binary comparisons, leading to low data utilization. To address this, researchers introduce RIMS, a three-stage preference optimization framework.

RIMS first generates synthetic chain-of-thought preference data via rejection sampling using only the target SLM, eliminating reliance on proprietary models. It then replaces hard selection with a differentiable soft aggregation mechanism that preserves gradient signal from all preference pairs while retaining the discriminative structure of margin-aware selection. Finally, it applies the smoothed objective to multiple alignment algorithms. The authors provide theoretical proof that the smoothed approximation admits a controllable error bound and yields tighter gradient alignment to the oracle objective than hard selection. Experiments on four multi-hop QA benchmarks demonstrate consistent gains in Exact Match and F1 under noisy retrieval conditions across multiple SLM backbones.