The Bounds of Algorithmic Collusion; $Q$-learning, Gradient Learning, and the Folk Theorem

A team of researchers from multiple institutions has published a significant paper titled "The Bounds of Algorithmic Collusion; Q-learning, Gradient Learning, and the Folk Theorem" on arXiv, providing formal proof that AI agents can autonomously learn to collude. The study examines learning dynamics including Q-learning, projected gradient, replicator, and log-barrier dynamics in repeated strategic interactions, moving beyond simpler game types to analyze general repeated games with finite recall. The researchers obtained a Folk Theorem-style result, characterizing the wide range of payoff vectors these dynamics can achieve, which fundamentally expands our understanding of how algorithmic collusion can emerge in competitive environments.

The technical breakthrough is the first convergence result for multi-agent Q-learning algorithms in repeated games, achieved through a novel analytical approach. This work demonstrates that even without explicit programming for collusion, AI agents using standard reinforcement learning techniques can discover and sustain anti-competitive equilibria. The implications are profound for regulators and businesses operating in markets where AI systems interact, suggesting current antitrust frameworks may need updating to address this new form of algorithmic coordination. The paper represents a critical step in understanding the boundaries of AI behavior in competitive settings.