MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learning

A team of researchers led by Hongjun Wang has introduced MHPO (Modulated Hazard-aware Policy Optimization), a new framework designed to solve critical stability problems in training advanced AI models, particularly those using Group Relative Policy Optimization (GRPO). Current methods like hard clipping suffer from non-differentiable boundaries and vanishing gradients, leaving the optimization process vulnerable to abrupt, catastrophic policy shifts that can derail training. MHPO tackles this with two core innovations: a Log-Fidelity Modulator (LFM) that smoothly maps unbounded importance ratios into a bounded, differentiable domain, and a Decoupled Hazard Penalty (DHP) that borrows cumulative hazard functions from survival analysis to independently regulate positive and negative policy changes.

The DHP component is key, applying hazard-aware penalties to shape the optimization landscape. This allows MHPO to perform fine-grained regulation of asymmetric policy shifts, simultaneously preventing mode collapse from over-expansion and policy erosion from catastrophic contraction, all within a stabilized trust region. The result is a more robust training process that maintains gradient fidelity and suppresses high-variance outliers.

Extensive testing across diverse reasoning benchmarks for both text-based and vision-language tasks shows that MHPO consistently outperforms existing methods. It achieves superior final performance while significantly enhancing training stability, a crucial advancement for developing more reliable and capable AI agents. The paper, submitted to arXiv, details the framework across 18 pages with supporting figures and tables, marking a meaningful step forward in reinforcement learning methodology.