Territory Paint Wars: Diagnosing and Mitigating Failure Modes in Competitive Multi-Agent PPO

Researcher Diyansha Singh has introduced 'Territory Paint Wars,' a new, open-source benchmark environment built in Unity designed to systematically diagnose why competitive multi-agent AI systems fail. The study uses this simple, symmetric zero-sum game to test Proximal Policy Optimisation (PPO) under self-play. Initially, an agent trained for 84,000 episodes achieved a dismal 26.8% win rate against a random opponent. Through controlled experiments, Singh pinpointed five critical, implementation-level bugs that contributed to this failure, including reward-scale imbalance, missing terminal signals, and incorrect win detection.

After fixing these bugs, the research uncovered a more insidious problem: competitive overfitting. Here, two co-adapting agents maintain a perfect 50/50 win rate against each other in self-play, creating the illusion of stable training. However, their ability to generalize to new, unseen opponents catastrophically collapses, with win rates plummeting from 73.5% to 21.6%. Crucially, this failure is invisible to standard self-play metrics. The paper proposes a remarkably simple yet effective solution called 'opponent mixing,' where 20% of training episodes use a fixed, random policy instead of the adaptive opponent. This minimal intervention, requiring no complex population-based training, successfully restored robust generalization, achieving a 77.1% win rate.

The work is significant because it provides a reproducible, minimal testbed for the multi-agent reinforcement learning (MARL) community. By open-sourcing the Territory Paint Wars environment, Singh offers a clear way to benchmark solutions against well-defined failure modes. This moves the field beyond ad-hoc fixes in complex environments like StarCraft or Dota 2, allowing for more rigorous diagnosis and mitigation of the fundamental pathologies that plague competitive AI training.