New AI Theory Explains Why We Make Bad Choices

This post introduces updatelessness to readers familiar with evidential and causal decision theory. Roughly, an updateless agent follows the policy she would have wanted to commit to before learning which situation she is in, even if a different action seems best after she updates on her observations.

The post uses examples where an agent would like to commit in advance to a policy she will later want to deviate from, introduces pseudo-updateless agents like son-of-EDT and son-of-CDT, and then describes full updatelessness, which requires no prior commitment and applies even to situations the agent is born into. It also gives an argument for updatelessness based on an example where combining EDT with updating leads to a clearly wrong choice, and it discusses philosophical ways of thinking about updatelessness, action versus policy updatelessness, and logical updatelessness.