Mean-Field Reinforcement Learning without Synchrony

Shan Yang's groundbreaking paper 'Mean-Field Reinforcement Learning without Synchrony' introduces the Temporal Mean Field (TMF) framework, solving a fundamental limitation in multi-agent AI systems. Traditional mean-field RL requires every agent to act simultaneously—an unrealistic constraint for real-world applications where agents operate on different schedules. The TMF framework replaces the problematic 'mean action' statistic with the population distribution μ, which remains defined regardless of which agents are active.

Technically, TMF operates across the full spectrum from fully synchronous to purely sequential decision-making within a single theory. The framework proves existence and uniqueness of TMF equilibria and establishes an O(1/√N) finite-population approximation bound that holds regardless of agent activation patterns. This represents a significant improvement over previous approaches that broke down when agents were idle. The accompanying TMF-PG (policy gradient) algorithm demonstrates convergence to the unique equilibrium.

Experiments on resource selection and dynamic queueing games confirm TMF-PG achieves near-identical performance whether one agent or all N agents act per step, with approximation error decaying at the predicted O(1/√N) rate. The framework's mathematical foundation includes exchangeability assumptions that ensure the population distribution fully determines each agent's reward and transition dynamics.

This breakthrough enables practical deployment of multi-agent systems in domains like autonomous vehicle coordination, robotic swarms, and distributed resource management where agents naturally operate asynchronously. The 21-page paper with 5 figures and 1 algorithm represents a major step toward scalable, real-world multi-agent intelligence.