Descent-Guided Policy Gradient for Scalable Cooperative Multi-Agent Learning

Researchers Shan Yang and Yang Liu have introduced Descent-Guided Policy Gradient (DG-PG), a breakthrough framework that fundamentally addresses the scalability limitations of cooperative multi-agent reinforcement learning (MARL). The core innovation tackles 'cross-agent noise'—the phenomenon where learning signals become increasingly noisy as more agents interact, causing gradient estimate variance to scale as Θ(N) with N agents.

DG-PG leverages domain-specific analytical models (common in cloud computing, transportation, and power systems) that prescribe efficient system states. By constructing noise-free per-agent guidance gradients from these differentiable models, DG-PG decouples each agent's learning signal from the actions of all others. The researchers proved this reduces gradient variance from Θ(N) to O(1), preserves game equilibria, and achieves agent-independent sample complexity of O(1/ε).

In practical testing on a heterogeneous cloud scheduling task, DG-PG demonstrated remarkable scale invariance. It converged within 10 episodes across all tested scales from 5 to 200 agents, directly confirming the theoretical predictions. Under identical architectures, established baselines like MAPPO and IPPO failed to converge. This represents a significant leap toward practical deployment of large-scale multi-agent AI systems in complex real-world environments where coordination among hundreds of intelligent agents is required.