GARIP: Running average reference stabilizes self-play in zero-sum games

In two-player zero-sum games, naive self-play causes gradient ascent cycles—the last iterate orbits the equilibrium instead of converging. Existing fixes like MMD use a fixed reference (reaching only regularized equilibrium) and R-NaD (from DeepNash) uses periodic snapshots. A new paper from Can Savcı introduces GARIP (Running-Average Moving Reference), which anchors training to the running average of past policies. The key insight: among causal convex averages with fixed mean lag, the running average's flat profile minimizes the peak lag—snapshots have a sawtooth pattern with peak = 2× mean. This reduces oscillation and enables local last-iterate convergence by scaling the base map's rotation by 1-β, effectively turning a recurrent system into a contraction.

Experiments across matrix games, the Coin Game, Connect Four, and Othello show GARIP matches R-NaD's peak robustness but is the better default hyperparameter: under conventional parameterizations, a matched-mean-lag GARIP collapses in 0/40 seeds vs 10/40 for R-NaD. The advantage emerges precisely where naive self-play cycles. The paper also explores anticipatory (negative-weight) references for stale-side improvements. All experiments are reproducible in pure JAX. This work offers a principled, drop-in improvement for training stable multi-agent systems without tuning periodic snapshot intervals.