Intrinsic Mutual Information as a Modulator for Preference Optimization

A new paper from researchers Peng Liao, Peijia Zheng, Lingbo Li, Shangsong Liang, and Lin Chen introduces RMiPO (Response-level Mutual information for Preference Optimization), a lightweight framework that addresses a key limitation of offline preference optimization methods like Direct Preference Optimization (DPO). While DPO and its variants are effective for aligning large language models (LLMs) with human values, they typically require extensive hyperparameter tuning to achieve optimal performance, leading to significant time overhead. RMiPO leverages intrinsic mutual information at the response level to dynamically decouple preference contributions, effectively modulating hyperparameters at negligible additional computational cost.

The experimental results demonstrate that RMiPO consistently outperforms prior methods across multiple benchmarks while reducing training overhead by more than 15%. This efficiency gain comes without sacrificing alignment quality, making it a practical improvement for teams deploying LLMs in production. The framework's lightweight nature means it can be integrated into existing training pipelines with minimal changes. The paper has been accepted at ACL Findings 2026, and the code is publicly available on GitHub. For AI practitioners focused on efficient model alignment, RMiPO offers a compelling path to reduce iteration time without compromising on performance or requiring manual tuning.