Claude Sonnet 4.6 and GPT-4.1 violate developer intent in 54-63% of code tasks
Both models pass 92% of tests yet miss hidden intent— a silent code quality crisis.
A new arXiv paper, DevIntent, exposes a critical blind spot in LLM code generation benchmarks: they measure only whether code passes given tests, not whether it respects the implicit intentions behind a developer's prompt. The researchers—Susana Haing, Natan Vidra, and Spurthi Setty—introduce the Intent Violation Rate (IVR) and a 49-problem pilot benchmark built from HumanEval+. Each problem strips implicit constraints from a clarified prompt and encodes them as hidden constraint tests that aren't visible to the model.
Evaluating Claude Sonnet 4.6 and OpenAI GPT-4.1, the results are striking: both models pass over 92% of visible stated tests, yet they violate underlying developer intent in over half of the problems—54.5% for Claude and 63.5% for GPT-4.1. The failures follow a systematic, bimodal pattern consistent across both models, indicating that pass rates significantly overstate how well generated code aligns with what developers actually want. The paper (arXiv:2608.07614) argues for intent-aware evaluation metrics and warns that relying on test-based benchmarks can silently ship code that behaves but misses crucial edge cases, such as unspoken security requirements or performance constraints. For teams using AI-assisted coding, this suggests that passing unit tests is far from sufficient for production-ready software.
- New benchmark: 49 HumanEval+ problems with hidden constraint tests to measure Intent Violation Rate (IVR).
- Claude Sonnet 4.6 violates intent in 54.5% of problems; GPT-4.1 does so in 63.5%, despite both passing 92%+ visible tests.
- Failures are bimodal and systematic across both models, meaning pass rates misleadingly overstate developer-intent alignment.
Why It Matters
Passing unit tests doesn't guarantee correct, intent-aligned code—teams need stricter validation for AI-generated software.