KatjaGrace on making AI extinction risk usable for decision-making
Most AI researchers put at least 5% on human extinction—yet global response is absent.
KatjaGrace, writing on LessWrong, tackles the disconnect between high subjective probabilities of AI-caused human extinction and the lack of a global response. She notes that most AI researchers assign at least 5% probability to such an outcome, with the most cited scientists (Yoshua Bengio and Geoffrey Hinton) putting it at 10% or more. However, these numbers are typically dismissed because they are subjective probabilities rather than frequentist frequencies derived from repeatable events.
Grace challenges this dismissal by pointing to real-world systems that already handle subjective probabilities well. Investors price a company's bankruptcy risk not from repeated experiments but from a collection of people's guesses about the situation. Similarly, medical prognoses for individual patients cannot rely purely on population frequencies. She suggests society could apply analogous mechanisms—like those used in finance or insurance—to make AI extinction risk an actionable input to global decision-making. The article is inspired by a talk from Paul Christiano and includes a comment from BryceStansfield arguing that the real issue is lack of salience, akin to the waning public concern over nuclear war despite increasing risks.
- Most AI researchers assign at least 5% subjective probability to advanced AI causing human extinction, with Bengio and Hinton at 10%+.
- Grace argues dismissal of these probabilities stems from their subjective nature, not from a lack of severity.
- She offers analogies: investors price bankruptcy risk via subjective guesses, and doctors personalize prognoses—showing subjective uncertainty can be made operational.
Why It Matters
Subjective expert risk assessments could become actionable if we adopt existing decision-making frameworks from finance and insurance.