New MARL Algorithm Reduces Variance with Clipping for Stable Convergence
Addresses exponential variance in advantage estimation for multi-agent systems
Cooperative multi-agent reinforcement learning (MARL) often uses trust region methods from single-agent settings. In the independent actors framework, each agent acts only on local information, which is more flexible but leads to exponentially high variance in the joint advantage function when agents update sequentially. This variance destabilizes convergence, even with importance sampling.
The authors propose a clipping objective that directly bounds the fluctuation of the advantage function during sequential updates. They prove a monotonic improvement bound and sub-linear convergence to ε-Nash equilibria. Two new algorithms based on this clipping objective are derived and tested on three standard MARL benchmarks. Results show superior performance over existing baselines, with stable convergence and significantly reduced variance. Code is publicly available for reproducibility.
- Novel clipping objective controls upper bounds of advantage fluctuation in sequential multi-agent updates.
- Provides theoretical guarantee of sub-linear convergence to ε-Nash equilibria with monotonic improvement.
- Outperforms tested baselines on three popular MARL benchmarks with stable convergence and low variance.
Why It Matters
Stable multi-agent learning enables scalable coordination in robotics, autonomous driving, and distributed control systems.