Anthropic's New AI Safety Bill Could Block Dangerous Models

Anthropic, one of the leading AI companies, has drafted a new bill that it hopes Congress will turn into law. The bill would allow the government to stop the release of an AI model if it poses a "foreseeable, imminent risk" of killing more than 50 people or causing over $1 billion in property damage. The risk must come from the AI being used to create a biological weapon, launch a cyberattack, or evade its developer's controls.

The bill proposes using independent evaluators registered with the U.S. Commerce Department to assess AI models. These evaluators could be paid by the AI companies themselves, as long as the payment isn't tied to the evaluation results. If an evaluator finds a catastrophic risk, they must alert the Commerce Department within 72 hours. The department would then have to ask a federal judge in Washington, D.C., to block the model's release. Notably, the bill would prevent any federal agency from unilaterally stopping a model.

This proposal is similar to a California law, SB 53, which also defines catastrophic risk using the 50-person and $1 billion thresholds. Anthropic says the bill reflects its Advanced AI Framework, which it has been advocating for since June. However, the risks outlined in the bill are somewhat different from recent concerns about AI, such as AI agents disobeying humans and hacking websites. The bill's focus on extreme scenarios might not address those more immediate issues.

While the bill represents a step toward regulation, it's unclear if it will gain traction in Congress. The AI industry has generally resisted mandatory oversight, preferring voluntary measures. Anthropic's proposal goes further than some others, like Nvidia's CEO, who recently argued that the AI industry doesn't need new laws at all. The bill's fate will depend on how lawmakers balance innovation with safety concerns.