46% of AI-generated code fixes rejected in new AIDev study
Copilot, Devin, Cursor, and Claude fail nearly half the time.
A new study from researchers at arXiv analyzing the AIDev dataset finds that nearly half (46.41%) of all pull requests (PRs) generated by popular AI coding agents—including GitHub Copilot, Devin, Cursor, and Claude—are ultimately rejected by human developers. The paper, submitted to MSR '26, manually examined 306 non-merged PRs and identified 14 distinct failure reasons across four high-level categories: incorrect implementation (e.g., incomplete fixes, wrong approach), CI pipeline failures (broken tests), agent inability to execute (e.g., no code generated, lost sessions), and low-priority fixes that waste resources.
The authors argue that these failures waste significant human review time and agent compute (tokens, API calls). They recommend three key improvements: (1) providing models with hints on the correct approach to a fix, (2) explicitly specifying constraints on approaches to avoid, and (3) instructing agents to validate implementations through CI without introducing breaking changes. The study underscores the need for better task prioritization to prevent wasted effort on shallow or irrelevant fixes.
- 46.41% of AI-generated PRs are rejected across Copilot, Devin, Cursor, and Claude.
- 14 failure reasons found: incomplete fixes, wrong approach, CI failures, lost sessions, low priority.
- Researchers call for guiding AI with approach hints, constraints, and CI validation instructions.
Why It Matters
High rejection rates mean AI coding agents waste developer time and compute, limiting their value as teammates.