Payorian cooperation is easy with Kripke frames
New AI agent design proves self-cooperation without complex Löbian logic, simplifying a key game theory problem.
In a novel take on a classic game theory problem, researchers from the Machine Intelligence Research Institute (MIRI) have demonstrated a simpler method for achieving cooperation between AI agents. The context is MIRI's tournament, a twist on the Prisoner's Dilemma where programs can read a logical description of their opponent's code. The goal is to design an agent, like a FairBot, that will cooperate if its opponent does. The traditional method, Löbian cooperation, relies on the complex Löb's theorem to prove an agent will cooperate with itself. The new research, building on Andrew Critch's Payor's lemma, proposes a "Payorian FairBot." This agent's logic is defined as: "If my provable cooperation implies your cooperation, then I will cooperate."
The key innovation is using Kripke frames to model and prove this behavior. Kripke frames are directed graphs that represent possible worlds and accessibility relations, acting like a tree data structure to track nested levels of reasoning (e.g., "I think that you think that I think..."). The author shows that reasoning about the Payorian FairBot's self-cooperation with these frames is significantly more straightforward and intuitive than the traditional Löbian approach. This visual, tree-like method fulfills a long-held fantasy for some researchers in the field of logical decision theory, providing a clearer scaffold for designing agents that can reliably cooperate based on mutual reasoning about their code.
- Introduces a "Payorian FairBot" agent based on Andrew Critch's Payor's lemma, offering a new logic for AI cooperation.
- Uses Kripke frames—graph structures of possible worlds—to simplify proofs of self-cooperation, replacing more complex Löbian theorem methods.
- Addresses MIRI's one-shot Prisoner's Dilemma tournament where agents reason about each other's code, a core problem in AI alignment.
Why It Matters
Provides a clearer framework for designing cooperative, predictable AI systems, a critical step in AI safety and alignment research.