CTRL-RAG: Contrastive Likelihood Reward Based Reinforcement Learning for Context-Faithful RAG Models

A research team led by Zhehao Tan has introduced CTRL-RAG, a new reinforcement learning (RL) framework designed to significantly improve the faithfulness of Retrieval-Augmented Generation (RAG) models. The core problem it addresses is that existing RL methods for RAG rely on external rewards, which often fail to accurately evaluate whether an answer is truly faithful to the retrieved documents. This can lead to models being rewarded for plausible but ungrounded answers, especially in open-domain settings. CTRL-RAG tackles this by proposing a hybrid "internal-external" reward system centered on a novel Contrastive Likelihood Reward (CLR).

The CLR works by directly optimizing the log-likelihood difference between the model's response when conditioned on a prompt with supporting evidence versus the same prompt without it. This internal reward mechanism incentivizes the model to extract and rely on the relevant context, increasing its confidence only when its answer is properly grounded. Experiments demonstrate that CTRL-RAG, used alone or combined with external correctness rewards, achieves strong results across various benchmarks testing reasoning and faithfulness. This approach directly combats the hallucination accumulation and potential model collapse that can occur with purely self-judging systems. The team has announced that training code and models will be released soon, providing a practical tool for developers building more reliable, evidence-based AI applications.