A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Critic

A research team from unnamed institutions has published a novel reinforcement learning algorithm called Hamiltonian Actor-Critic (HAC) on arXiv. The work, led by Chengyang Gu, Yuxin Pan, Hui Xiong, and Yize Chen, addresses a fundamental problem in model-based RL: compounding errors from imperfect learned dynamics models that degrade long-term planning. Traditional actor-critic methods and improvements like Model-Based Value Expansion (MVE) remain sensitive to rollout horizon selection and residual model bias, limiting their reliability.

HAC's innovation comes from applying the Pontryagin Maximum Principle (PMP), a cornerstone of optimal control theory developed in the 1950s. Instead of learning an approximate value function—a common source of error—HAC directly optimizes a Hamiltonian function defined over the learned environment model and reward. This approach sidesteps the error propagation issue and provides stronger theoretical convergence guarantees for deterministic systems. The authors report that HAC demonstrates superior performance on continuous control benchmarks compared to both model-free and MVE-based baselines.

The algorithm shows marked improvements in three key areas: final control performance, speed of convergence, and robustness to distributional shift, including challenging out-of-distribution (OOD) scenarios. Perhaps most notably, in offline RL settings where agents must learn from a fixed, limited dataset without further interaction, HAC matched or exceeded state-of-the-art methods. This highlights its exceptional sample efficiency, a critical metric for applying RL to real-world systems like robotics where data collection is expensive or risky. The 18-page paper includes extensive experiments validating these claims across different task domains.