KL-divergence gating lifts multi-agent RL success by 11 points

Multi-agent reinforcement learning (MARL) systems face a classic trade-off: communicate too often and you waste bandwidth; communicate too rarely and you lose coordination. Existing solutions either broadcast at every timestep or learn a binary gate via REINFORCE policy gradients, which suffers from high variance and unstable behavior. Teoman Kaman's new paper proposes a principled alternative: each agent maintains a belief distribution over a latent world state (computed as a softmax over its LSTM hidden state), and triggers communication only when the KL divergence between agents' beliefs exceeds a fixed threshold. This makes the gating decision interpretable, stable, and based on actual disagreement rather than a noisy learned signal.

Results show the approach shines when coordination is hardest. On the Predator-Prey 10x10 benchmark, IC3Net still edges out the KL-belief method at all thresholds. But on the more demanding 20x20 grid, a threshold ablation reveals an inverted U-shape: epsilon=0.5 achieves 73.84 average steps and 42% success versus IC3Net's 75.31 steps and 31% — an 11-point boost in success with tighter seed variance. On MPE simple_spread, the belief head improves mean reward by 12 points and reduces variance by 26x even when gating is inactive, suggesting two independent benefits: principled communication gating when beliefs converge, and richer latent representations that help coordination regardless. The paper provides a strong case for belief-based gating as a lightweight, interpretable alternative to learned communication policies.