Byrnes reconciles AI alignment debate: LLMs won't scale to ASI

In a post on the AI Alignment Forum, Steven Byrnes presents a nuanced take on the heated debate between Yudkowsky & Soares and the broader LLM research community. The former camp believes that without fundamental breakthroughs in alignment, future ASI will inevitably become scheming, out-of-control rogue agents. The latter camp, consisting of many LLM practitioners, sees current techniques like RLHF and fine-tuning as largely adequate and expects them to remain so as models improve. Byrnes validates both perspectives by arguing that they apply to different architectures: the dire predictions about misalignment are correct for hypothetical ASI, but LLMs themselves won't evolve into such systems. He suggests that LLMs will hit a ceiling short of general superintelligence, thereby sidestepping the worst-case alignment scenarios. This position challenges the common assumption that scaling LLMs alone leads to AGI and implies that other breakthroughs—perhaps new architectures or hybrid systems—would be needed to achieve superintelligence. Byrnes also critiques both sides: Yudkowsky & Soares for overstating the intractability of alignment and ignoring the relevance of LLMs, and LLM researchers for dismissing the theoretical concerns about ASI as armchair speculation. The post concludes with sympathy for both camps, advocating for humility and a broader view of the AI landscape.