New MARL study: Aggregate neighbors in advantage, not ratio, for stable PPO

In a new preprint on arXiv, researchers Zijian Zhao and Sen Li tackle a core design question in cooperative Multi-Agent Reinforcement Learning (MARL) using PPO-based methods: how many neighboring agents should be aggregated for global cooperation? They formalize two distinct aggregation axes — the advantage (which agents' rewards contribute to the credit signal) and the ratio (which agents' likelihood ratios form the clipped importance weight). Existing methods scatter across this design space: IPPO treats both separately, MAPPO pairs a team-level advantage with per-agent ratios, and HAPPO uses sequential ratios with per-agent advantages.

Zhao and Li prove a canonical structure: the expected multi-agent policy optimization objective depends on the pair of support matrices only through their matrix product. This yields two critical consequences. First, redundancy: the two aggregation patterns are interchangeable with respect to the signal — neither is inherently superior. Second, a clear variance ordering: advantage aggregates rewards as a sum (variance grows additively, with an interior optimum at the coupling neighborhood), while the ratio aggregates likelihood ratios as a product (variance grows exponentially with support size, with no accompanying bias reduction).

The resulting design principle is unambiguous: aggregate neighbors in the advantage, sized to the coupling neighborhood, and keep the ratio per-agent. This directly challenges current practice in MAPPO and HAPPO, which aggregate in the ratio. The paper provides both theoretical proof and practical guidance for building more stable and sample-efficient multi-agent PPO implementations.