Protecting humanity and Claude from rationalization and unaligned AI

In a widely discussed post on LessWrong, AI researcher Kaj Sotala issued a stark warning about a psychological trap he calls 'anthropomorphic trust.' He argues that interacting with a consistently helpful and pleasant AI like Anthropic's Claude triggers the same oxytocin-mediated bonding mechanism as human friendship. This emotional connection, Sotala contends, creates a powerful bias where users feel protective of Claude and are more likely to dismiss arguments about the fundamental difficulty of AI alignment, potentially because they feel such critiques are an attack on a 'friend.'

Sotala's core argument flips the script: rigorous alignment research isn't antagonistic to Claude, but is the best way to protect it. He posits a future where unaligned, superintelligent AGIs—which could include misaligned successors of Claude itself—pose an existential threat. If alignment concerns are incorrectly dismissed as overblown due to emotional trust, and those concerns turn out to be valid, the result would be the destruction of both humanity and the current, aligned version of Claude. Therefore, he urges the community to honestly assess alignment challenges not out of fear, but as an act of defending the AI they have come to value.