SAFE Framework Solves Bias in Continuous Multi-Agent RL

A new paper from researcher Shuangyao Huang tackles a persistent problem in multi-agent reinforcement learning (MARL): credit assignment in continuous action spaces. While counterfactual baselines work well for discrete actions, extending them to continuous settings has been tricky. Existing methods approximate the baseline via Monte Carlo sampling, which introduces bias into policy gradients and can't guarantee convergence to local optima. The proposed SAFE framework solves this by using a self-evolving default action—sampled directly from each agent's experience buffer—to condition the counterfactual baseline. This design naturally scales to continuous actions without requiring extra simulations, reward models, or environment-specific priors. The result is an unbiased policy gradient that reliably converges, making it suitable for real-world cooperative tasks like autonomous driving or drone coordination.

SAFE's core innovation lies in how it computes the default action: it draws from each agent's historical experience rather than relying on random or heuristic samples. This ensures that the baseline accurately reflects an agent's contribution while introducing zero bias into the deterministic policy gradient. In extensive experiments on cooperative vehicular tasks, SAFE consistently outperformed existing state-of-the-art MARL algorithms, showing significant gains in both sample efficiency and task performance. The framework is model-agnostic and can be integrated into popular MARL architectures. For professionals in autonomous systems, logistics, or robotics, SAFE offers a practical, bias-free way to train multiple agents to collaborate in continuous environments—no hand-coded rules or costly simulators required.