New study: Independent multi-agent learning beats centralized in embodied gridworld

A new arXiv preprint (2601.17454) from Muhammad Ahmed Atif and colleagues challenges a core assumption in multi-agent reinforcement learning (MARL): that centralized value learning is inherently superior to independent learning. The team isolated coordination structure from function approximation and partial observability using a fully tabular 8x8 predator-prey gridworld with explicit speed and stamina constraints. They tested all four pairings of Independent Q-Learning (IQL) and Centralized Q-Learning (CQL) across three kinematic regimes over 10 random seeds.

Results were stark: fully independent agents (IQL-IQL) outperformed fully centralized ones (CQL-CQL) in every regime and every seed, with a Wilcoxon p-value of 0.00195 and a Cliff's delta of 1.0 — a perfect effect size. Mixed IQL-CQL configurations produced coordination breakdowns that persisted for the full 40,000-episode training budget, rather than resolving as transient instability. The authors propose a mechanism called 'temporal synchronization lock': because a centralized value function couples all agent decisions, when one agent is stamina-limited, the joint Q-value forces capable partners into suboptimal waiting. Independent learners, by contrast, can continue asynchronous pursuit. This pathology arises purely from credit assignment, not function approximation, so analogous effects may occur in deep MARL methods that centralize credit during training.