ARMS: First automatic reward shaping framework preserves Nash equilibria in multi-agent RL

Sparse rewards have long plagued multi-agent reinforcement learning (MARL), where simultaneous learning introduces non-stationarity that makes manual reward design brittle and often counterproductive. In a new paper titled "ARMS: Automatic Reward Shaping for Sparse-Reward Multi-Agent Reinforcement Learning," researchers Elie Abboud and Oren Gal from the Technion propose ARMS, a self-supervised reward shaping framework that learns dense shaping signals from sparse environmental rewards through trajectory ranking. The core theoretical contribution is a reformulation of policy invariance via conditional best-response reasoning: if certain conditions hold, the shaping rewards preserve each agent's best-response set under fixed opponent policies, and consequently preserve the set of Nash equilibria—a game-theoretic guarantee absent from prior automatic reward shaping approaches. ARMS alternates between policy learning and reward learning, sharing shaping parameters across agents for efficiency.

Empirically, ARMS was tested in partially observable multi-agent pathfinding domains, outperforming baselines in sampling efficiency under increasing reward sparsity and agent count. The framework also generalized to unseen environments without retraining. Notably, the authors identified a MARL-specific failure mode: limited exploration coupled with coupled policy–reward dynamics can induce oscillatory behavior, where agents cycle through suboptimal strategies. Increasing exploration effectively stabilizes learning. ARMS is, to the authors' knowledge, the first automatic reward shaping framework for MARL motivated by an equilibrium-preservation result, potentially enabling more reliable training of collaborative and competitive multi-agent systems in robotics, autonomous driving, and game AI.