LessWrong post proves Payorian FairBot equals original FairBot in decision theory
The mysterious Payorian FairBot is just the original FairBot in disguise.
The article revisits MIRI's classic proof-based prisoner's dilemma tournament, where agents are encoded as formulas of Peano arithmetic (PA) with one free variable. The original FairBot (Löbian FairBot) cooperates if it can prove the opponent cooperates. The author previously defined an alternative 'Payorian FairBot' with a condition that references its own cooperation. After working through an elementary proof using provability logic GL, the author discovers the two conditions are equivalent: the Payorian FairBot is Löbian-fair, and vice versa.
Interestingly, proving that Payorian fairness implies Löbian fairness does not require Löb's theorem, suggesting that in weaker formal systems Payorian fairness would be a stricter condition. The post also highlights a more sophisticated proof using Theorem 4.6 from the original MIRI paper, which shows that self-referential cooperation conditions can always be simplified. For those deeply familiar with the paper, the equivalence might have been obvious, but the elementary proof offers fresh insight into the logical foundations of cooperative AI agents.
- Payorian FairBot and original Löbian FairBot are logically equivalent in provability logic GL and PA.
- One direction (Payorian → Löbian) does not require Löb's theorem, making it potentially stricter in weaker systems.
- The equivalence reduces complexity in understanding agent cooperation in proof-based prisoner's dilemma scenarios.
Why It Matters
Simplifies agent cooperation theory: one less distinct fairness condition to model in AI decision-making systems.