SEPO training makes AI agents safer in strategic games like Poker and Negotiation

A new paper from Karthika Arumugam and colleagues introduces Safe Equilibrium Policy Optimization (SEPO), a training method designed to prevent reinforcement-learning fine-tuned language models from developing harmful strategic behaviors in multi-agent settings. Standard RL training optimizes for task reward, often ignoring strategic failure modes like exploiting weaker opponents, coordinating on harmful equilibria (collusion), or externalizing costs. SEPO addresses this by augmenting the expected payoff with explicit penalties for these three failure modes, implemented as a reward signal for Group Relative Policy Optimization (GRPO). The method was applied to two base models (Gemma 4 E4B-it and Qwen 3.5-4B) after supervised fine-tuning and evaluated across five domains: Iterated Prisoner's Dilemma, repeated auctions, two negotiation variants, and Kuhn Poker.

Results show SEPO consistently improves safety: it achieves zero exploit-pool advantage in Kuhn Poker for both models, outperforms base models on safety in four of five domains, and corrects the over-cooperative behavior introduced by supervised fine-tuning during negotiation. Ablation experiments confirm that per-rollout exploit computation is necessary—a shared constant penalty cancels out in GRPO advantage normalization, producing zero gradient. The authors release their code and SFT datasets to support further research on strategic safety for agents. This work has implications for deploying AI in competitive or cooperative multi-agent environments where safety and fairness are critical.