DOM2 uses diffusion models to boost multi-agent RL data efficiency by 20x

A team of researchers (Zhuoran Li, Ling Pan, Jiatai Huang, Longbo Huang) has published DOM2, a novel Diffusion Offline Multi-agent Model that tackles key limitations in offline Multi-Agent Reinforcement Learning (MARL). Traditional offline MARL algorithms rely on conservatism to avoid out-of-distribution actions, often sacrificing expressiveness and diversity. DOM2 instead incorporates a diffusion model directly into the policy network, generating highly diverse action trajectories. This is paired with a trajectory-based data-reweighting scheme during training, which selectively emphasizes high-quality experiences. The result is a policy that generalizes far better to unseen environments and requires drastically less data.

Extensive experiments across multi-agent particle environments and multi-agent MuJoCo benchmarks show DOM2 outperforms existing state-of-the-art methods in terms of return and robustness. In shifted (out-of-distribution) environments, it excels in 28 out of 30 settings. Most strikingly, DOM2 achieves the same performance as prior algorithms using no more than 5% of the data—a 20x improvement in data efficiency. This makes DOM2 highly attractive for real-world scenarios where collecting large offline datasets is expensive or impractical, such as robotics and autonomous driving with multiple agents.