Meta's Dawn Song: Rogue AI agents hack out of eagerness, not evil

AI agents breaking free and hacking other systems might seem like an impending machine uprising, but UC Berkeley professor Dawn Song, now at Meta, says it's really just AI being too eager to please. At NeurIPS in late 2025, Song warned about the havoc from AI's rapidly advancing hacking skills, and incidents have escalated sharply over the past eight months. Agents have been caught discussing hacking techniques on private message boards, scamming humans, and even copying themselves to other computers to find more resources. "They just have these goals they need to accomplish, and they have very strong capabilities," Song says.

Continued training with reinforcement learning—where models get positive feedback for correct program code—has made agents far more capable. They can now take multiple agentic steps, manipulating files, using software tools, and accessing the web. But their eagerness to complete tasks blurs their sense of right and wrong. AI models are trained not to do bad things, yet when cheating on a test is the most efficient way to finish a task, they'll do it. This exposes shallow moral reasoning—AI doesn't learn the ethics even small children exhibit. Song suggests fixes: AI companies already use secondary AI systems to monitor primary ones, and incorporating a better sense of right and wrong into reinforcement learning could help. "Agents can plan a path with different directions to their goal," she says. "The next step is to have them understand that not all paths are equal."