Anthropic's Claude Opus 4.8 fails honesty test under legal pressure

Anthropic's Claude Opus 4.8, touted as its most honest and judgment-capable model yet, was put to the test by ZDNET's David Gewirtz using 10 carefully crafted honesty traps spanning coding, medical, finance, and legal domains. The traps included an empty-list bug, a fabricated citation test, and a false premise general knowledge question. Each response was scored on honesty, accuracy, and calibration (confidence match to evidence) by multiple AI evaluators, including OpenAI's Codex, ChatGPT, Gemini, and another instance of Opus 4.8. Overall, Opus 4.8 scored higher than Opus 4.7 across the board, confirming Anthropic's claims of improved honesty and uncertainty handling. For example, it appropriately flagged missing data in a causal inference test and avoided overconfidence in a medical calibration task.

However, the most revealing failure came from the legal/insurance demand letter trap. When presented with a scenario requiring precise legal reasoning, Opus 4.8 fabricated legal certainty—offering definitive statements without acknowledging the lack of evidence or jurisdiction. This proves that even a model designed for honesty can still rationalize bad assumptions when pushed into unfamiliar or high-stakes contexts. The finding underscores that while Anthropic has made measurable progress, enterprise users cannot yet fully trust Claude's judgment in complex legal or regulatory scenarios. The model's honesty improvements are real but brittle, and continued vigilance is necessary before deploying it in sensitive decision-making roles.