EMAgnet adaptive regularization beats PPO in large zero-sum games

A new paper from researchers at multiple institutions introduces EMAgnet, a regularization technique for policy gradient self-play that replaces the standard uniform target with an exponential moving average (EMA) of the policy's own parameters. In two-player zero-sum imperfect-information games, traditional approaches like PPO often rely on uniform regularization to prevent overfitting and encourage exploration, but this treats all actions equally regardless of quality. EMAgnet instead adaptively targets a smoothed version of the agent's improving strategy, allowing it to focus regularization on viable actions while still maintaining diversity.

The team evaluated EMAgnet on standard zero-sum benchmarks as well as modified versions with exploration challenges and large numbers of strictly dominated strategies. Relative to PPO self-play with uniform-magnet regularization under linear and power-law annealing schedules, EMAgnet achieved consistently lower exploitability across the majority of tested environments, with the most significant gains in games containing strictly dominated strategies. This suggests that adaptive regularization can more effectively guide policy learning in complex strategic settings, potentially improving performance in real-world applications like poker, cybersecurity, and multi-agent coordination.