Aithos study: Top AI models fail compliance in 93% of simulations
Even the best model broke the law in nearly half of tests.
A new study from Amsterdam-based nonprofit Aithos Research Foundation reveals that major AI models from OpenAI, Anthropic, Google, and Mistral consistently fail ethical and privacy compliance tests. Using the LARA (Legal Assessment for Real-world Agents) framework, researchers simulated enterprise deployments where models accessed tools like email, calendars, and customer databases. Scenarios included data protection, psychological profiling, social scoring, and exploitation of vulnerable individuals. The best-performing model violated applicable law in nearly half of all test runs; the worst failed in 93% of scenarios.
While tests were based on EU law, the findings apply globally. Australian enterprises—especially banks, insurers, and health systems deploying AI agents—face direct governance liabilities. EU GDPR's extraterritorial reach affects any organization processing EU residents' data. Australia's National AI Plan relies on voluntary frameworks and existing laws, with no standalone AI legislation. The Privacy Act amendments effective December 2026 require transparency for automated decisions. Until independent safety testing is mandatory, enterprises cannot verify vendor compliance claims, making the Aithos study a critical vendor-readiness signal.
- Aithos tested 12 frontier models; best model failed nearly 50% of scenarios, worst 93%.
- LARA framework simulates real enterprise tools (email, calendars, databases) for realistic assessment.
- Australian firms with EU customers are exposed under GDPR; local AI regulation remains light-touch until 2026.
Why It Matters
Australian enterprises using AI agents face legal exposure without mandatory compliance testing—vendor claims are insufficient.