Why iterative AI development may defuse classic superintelligence takeover scenarios

The LessWrong piece 'Superintelligence of the gaps' by vals tutor reexamines classic AI doom scenarios that assume a superintelligence suddenly emerges with vastly superior strategic abilities. The author argues this hinges on a large intelligence gap between the current best system and the new one—a gap that is technically easy to avoid. In practice, leading AI labs deploy models iteratively, evaluate at regular checkpoints, and supervise new systems with the combined resources of prior aligned models. This gradual approach prevents any single model from being significantly smarter than its predecessors, making fast, misaligned takeovers unlikely.

Commenter Vladimir_Nesov pushes back, noting that superintelligence is defined by qualitatively outperforming humanity—not just the previous AI. Even without gaps between models, a superintelligence could emerge from humans wielding weaker AIs or from a breakthrough in learning. The original author concedes this point but maintains that the leading companies will be cautious with powerful systems, citing the 'AI 2027' scenario where premature deployment—not technical impossibility—causes failure. The bulk of the author’s p(doom) therefore shifts to gradual disempowerment: a slow erosion of human control through accumulated AI influence. This scenario offers more time for solutions, but no clear path yet.