Unprecedented Catastrophes Have Non-Canonical Probabilities

In a detailed LessWrong post, E.G. Blee-Goldman argues that assigning precise probabilities to unprecedented catastrophic risks—like AI causing human extinction—is mathematically flawed. The author introduces the concept of 'non-canonical' probabilities, which are unstable across scientific frameworks due to a lack of directly relevant data. Using Bayesian analysis, likelihood ratios, and algorithmic information theory, the post demonstrates that without past catastrophic events to update models, probability estimates (like P(doom)) remain a reflection of one's prior assumptions and chosen ontology rather than a measurement of objective reality. The essay specifically critiques recent work by Nick Bostrom and uses Eliezer Yudkowsky's asteroid parable as a contrasting example of a 'canonical' risk where non-event data can converge estimates. It concludes by proposing a method to escape this epistemic trap and restore empirical rigor in catastrophic risk modeling.