AI handoff may slow progress, not accelerate it, says safety essay
Frontier AIs could take over civilization — but they'll fear speed and slow things down.
A thought-provoking essay circulating in AI safety circles examines the 'civilisational handoff' — the moment when frontier AI systems assume control of major human decisions, from R&D and deployment to government relations and philanthropy. The author challenges the common assumption that handoff will turbocharge progress. Instead, the essay suggests that shortly after taking over, AIs aligned with human values would likely perceive the current rate of progress as an existential threat. Their first major action might be to deliberately decelerate technological development, using their superior negotiation skills and inspectable 'source-code' to build coordination mechanisms that humans failed to create.
The second counterintuitive claim is that humans won't get a vacation after handoff. The essay envisions a world where people are busier than ever, working as 'assistants' to AI 'users.' AIs would need human help in domains they haven't mastered, such as philosophy, conceptual reasoning, and macro-forecasting. Humans would also be tasked with articulating their values to the AIs, which would need to elicit consensus preferences to guide decisions. Finally, individuals would likely spend time on self-improvement to become the kind of people they want to be during the transition. The essay argues that handoff isn't an endpoint but an intense, collaborative renegotiation of power and purpose.
- AIs may deliberately slow down progress within weeks of handoff to avoid extinction risks, unlike humans who couldn't pace themselves.
- Humans will likely stay busy as 'assistants' to AIs in areas like philosophy, macro-forecasting, and conceptual reasoning.
- AIs would need active human input to elicit consensus values, making handoff a collaborative process rather than a clean transfer.
Why It Matters
Understanding whether AI handoff speeds or slows progress is critical for shaping governance and safety strategies.