Yudkowsky's Rationality Sequences: Preface Mistakes & Bias Basics

The preface of Eliezer Yudkowsky's 'Rationality: From AI to Zombies' offers a candid self-assessment of his earlier Sequences. He identifies five major mistakes: overemphasizing theoretical rationality over practical day-to-day application, focusing on grand philosophical issues rather than everyday problems, prioritizing rational belief over rational action, chaotic post organization, and being too dismissive of opposing ideas. The core mission remains teaching skills of rational belief and decision-making, which modern education neglects. The introduction, by Rob Bensinger, delves into cognitive biases as systematic errors in thinking, distinguishing them from statistical biases. It introduces Kahneman and Tversky's dual-process theory: System 1 (intuitive, automatic) and System 2 (deliberate, slow). Key biases like the representativeness heuristic, conjunction fallacy, and base rate neglect are explained. Importantly, mere awareness is insufficient due to the blind spot bias — we easily see biases in others but not ourselves. Acquiring expertise involves understanding the rationale behind biases, spotting them in specific contexts, and developing countermeasures.

The author, manueldelrio, shares personal reflections, questioning the difference between mistakes #1 and #3 (theory vs. action). They express skepticism about the assumed tractability of molding human rationality into a coherent system, viewing humans as 'clunky Rube Goldberg machines' with deeply inconsistent preferences from evolutionary pressures. This places a low upper bound on achievable rationality without radical genetic or technological modification. The post serves as a thoughtful entry point for readers embarking on the 1,750-page journey through the Sequences. It underscores both the ambition and limitations of Yudkowsky's rationality project, inviting critical engagement rather than blind acceptance. For AI safety enthusiasts and rationality practitioners, this meta-review highlights the importance of practical application and humility in overcoming cognitive biases.