Cheap Open Models Reportedly Reproduced Much Of Mythos's Showcased Findings

A new analysis from Aisle.com has sent ripples through the AI security community by demonstrating that small, inexpensive open models can match the core cybersecurity findings of Anthropic's specialized and heavily marketed 'Mythos' model. The team took the specific vulnerabilities Anthropic showcased—including a flagship FreeBSD exploit and a 27-year-old OpenBSD bug—and tested them against a suite of open-weights models. The results were striking: all eight tested models, including a tiny 3.6-billion-parameter model costing just $0.11 per million tokens, successfully detected the FreeBSD exploit. Furthermore, a 5.1-billion-parameter open model recovered the core exploit chain for the decades-old OpenBSD vulnerability. This directly challenges the narrative that cutting-edge cybersecurity requires massive, proprietary frontier models.

The findings suggest the AI cybersecurity capability frontier is 'jagged,' meaning performance doesn't scale predictably with model size or cost, and there is 'no stable best model' across different security tasks. Critics, including Meta's Yann LeCun, have seized on the report to label the Mythos launch as 'marketing/hype,' noting that Anthropic's testing harness—which prompted the model separately for each file—may have been a significant factor in its performance, a methodology not applied to test existing models like Claude 3 Opus. The implication is that the true 'moat' in AI security may be the expert-built system and harness around the model, not the underlying architecture itself. While Mythos validates a focused approach to AI security, this independent testing indicates the claimed architectural breakthrough might be less revolutionary than presented, potentially making sophisticated security analysis more accessible and affordable sooner than expected.