TPMM-DPO: New method merges models to fix DPO training instability

Direct Preference Optimization (DPO) is a popular method for aligning large language models without an explicit reward model, but its iterative variant has a critical flaw: using the previous iteration's policy as the reference model causes noise and errors to accumulate over time, leading to late-stage over-optimization, performance fluctuations, and degraded generalization.

To address this, researchers from an academic team (paper by Lingling Fu and Yongfu Xu) introduce TPMM-DPO (Trajectory-aware Preference-guided Model Merging for Iterative Direct Preference Optimization). The core idea is to treat the sequence of policy models generated during iterative DPO as an optimization trajectory. Instead of relying on a single previous model as the reference, TPMM-DPO adaptively integrates multiple past models using learned fusion weights. This creates a smoother, more robust reference model that mitigates error accumulation from noisy preference data. Experimental results show that standard iterative DPO suffers from performance degradation in middle and later training stages, while TPMM-DPO consistently improves generation quality, achieving higher win rates and reward scores. Ablation studies confirm that learnable-weight fusion outperforms simple averaging, especially in late-stage performance stability.