Oliver Sourbut warns automated AI R&D could concentrate power dangerously

The prospect of automated AI research and development has been discussed since I.J. Good's 1965 'intelligence explosion' hypothesis, but it became a public talking point in late 2025. OpenAI CEO Sam Altman set a target of fully automated AI-improving-AI by 2028, and Anthropic co-founder Jack Clark stated that 'AI systems are about to start building themselves.' Philosopher Oliver Sourbut, in a LessWrong post, argues that the most important effect of this trend is not speed, but concentration of influence. He points out that fewer human participants in the R&D loop means fewer whistleblowers, less internal scrutiny, and weaker decision-making robustness. This creates more single points of failure, where individuals or small groups become vulnerable to capture, coercion, or rash decisions—even if they are initially wise.

Sourbut emphasizes that the concentration of influence raises incentives for corruption and power struggles, while simultaneously increasing the stakes for everyone else. He notes already-visible turf wars within AI companies, boardroom battles, and clashes with governments. The roles most likely preserved longest are senior research directors, executives, and closely-involved government officials—meaning those lost are the broader workforce's checks and balances. Sourbut warns that losing this oversight is a problem regardless of who is at the top, and that the centralization of AI production means the centralization of economic power more broadly. He calls for paying close attention to leading indicators of this concentration.