CDCPG algorithm cracks continuous multi-agent RL with local processing

A team of researchers from Southeast University and UC Riverside introduced the Continuous Distributed Coupled Policy Gradient (CDCPG) algorithm, addressing the longstanding challenge of scaling multi-agent reinforcement learning (MARL) to continuous state-action spaces in networked systems. Unlike prior approaches that suffer from exponential growth in complexity or require full network information, CDCPG exploits the spatial decay of interactions: each agent maintains an actor over a bounded graph neighborhood and a localized least-squares temporal-difference critic that evaluates a truncated action-value function. The critic uses a spectral random-feature representation of the local transition kernel, enabling a well-posed localized Bellman theory that avoids naive truncation errors.

The paper makes four key theoretical contributions. First, it constructs the truncated action-value function as a conditional expectation over the neighborhood, removing the continuation-kernel mismatch. Second, it exposes a dimensional obstruction in temporal-difference stability for normalized random features and proves an unconditional excitation bound, reducing stability to a symmetric persistence-of-excitation condition monitorable via online matrix concentration. Third, under exponential spatial decay of agent interactions plus the excitation condition, CDCPG drives the averaged per-agent stationarity measure to within any ε of an explicitly characterized approximation floor using Õ(ε⁻²) shared-oracle samples—matching the smooth nonconvex first-order rate. Fourth, an adaptive-locality rule selects the neighborhood radius that balances truncation and graph-decay residuals against target accuracy. Simulations on a networked linear-quadratic benchmark validate the predictions on locality and feature dimension.