Risk-seeking conservative policy iteration with agent-state based policies for Dec-POMDPs with guaranteed convergence

A team of researchers including Amit Sinha, Matthieu Geist, and Aditya Mahajan has introduced a novel algorithm for optimizing decentralized multi-agent AI systems, formally known as Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs). Finding optimal solutions for these systems, where multiple agents must cooperate with limited information, is notoriously difficult (NEXP-complete). The new method addresses a key practical constraint: limited memory. Instead of policies based on an agent's full history, it uses compact 'agent-state' policies, which model a limited number of internal memory states. The core innovation is an 'iterated best response' style algorithm combined with a modified objective that incentivizes risk-seeking behavior alongside a conservative policy update. This combination guarantees that the solution improves monotonically and converges to a local optimum, all within a polynomial runtime relative to the problem's size—a significant theoretical and practical advance.

Empirical tests on standard Dec-POMDP benchmarks show the algorithm performs as well as current state-of-the-art methods, achieving near-optimal results despite the strict memory limits. The research also demonstrates a clear performance trade-off: using more agent states (a larger memory allowance) leads to better outcomes, giving engineers a direct lever to balance capability against computational cost. This work provides a crucial new framework for incorporating realistic memory constraints directly into the training and deployment of collaborative multi-agent AI. It opens the door for more efficient AI teams in applications like autonomous vehicle coordination, robotic swarms, and networked sensor systems, where processing power and memory are often at a premium.