NePPO: Near-Potential Policy Optimization for General-Sum Multi-Agent Reinforcement Learning

A team from UC Berkeley led by Addison Kalanther, Sanika Bharvirkar, Shankar Sastry, and Chinmay Maheshwari has introduced NePPO (Near-Potential Policy Optimization), a novel algorithm designed to tackle one of multi-agent reinforcement learning's toughest challenges: finding stable equilibria in general-sum games where agents have mixed cooperative and competitive objectives. Unlike existing methods that struggle with unstable learning dynamics or only work in restricted settings like zero-sum or fully cooperative games, NePPO introduces a clever mathematical workaround. It learns a single, player-independent potential function—essentially a shared utility approximation—that transforms the complex original game into a more tractable cooperative version. By minimizing a novel objective function using zeroth-order gradient descent, the algorithm identifies policies that approximate Nash equilibria of the original mixed-motive environment.

Empirical results demonstrate NePPO's practical superiority over popular MARL baselines including MAPPO, IPPO, and MADDPG. The algorithm's core innovation lies in its pipeline that systematically searches for the best potential function candidate, which then guides agents toward approximate equilibrium policies. This approach addresses the fundamental problem in general-sum MARL: when agents have heterogeneous and potentially conflicting preferences, there's no clear system-level objective to optimize. By constructing this proxy potential function, NePPO provides the missing coordination mechanism, enabling more stable convergence in environments ranging from autonomous vehicle coordination to economic simulation and robotic team interactions where cooperation and competition naturally coexist.