Constraint Decay study: LLM agents lose 30 points on backend code

A new arXiv preprint from researchers Francesco Dente, Dario Satriani, and Paolo Papotti systematically evaluates the fragility of LLM agents when generating backend code under real-world constraints. The study, titled 'Constraint Decay: The Fragility of LLM Agents in Backend Code Generation,' introduces a dual evaluation methodology combining end-to-end behavioral tests with static verifiers. By fixing a unified API contract across 80 greenfield generation tasks and 20 feature-implementation tasks spanning eight web frameworks (Flask, FastAPI, Django, etc.), the team isolated the effect of structural complexity. Their core finding: LLM performance degrades sharply as structural requirements accumulate—a phenomenon they term 'constraint decay.' Baseline assertion pass rates dropped by an average of 30 points when moving from loose to fully specified tasks, with some weaker configurations approaching zero pass rates.

The study also reveals significant framework sensitivity. Agents succeed on minimal, explicit frameworks like Flask but perform substantially worse on convention-heavy environments like FastAPI and Django, where object-relational mappings and architectural patterns are enforced. Error analysis pinpoints data-layer defects (incorrect query composition and ORM runtime violations) as the leading root cause of failures. This work highlights that jointly satisfying functional and structural requirements remains a key open challenge for coding agents, especially as production-grade software demands strict adherence to non-functional specifications often overlooked in existing benchmarks.