Study: AI agent instructions boost merge rates for only 27.7% of projects
New study of 15,549 agentic PRs shows instruction files are no silver bullet.
A new study from researchers Ali Arabat and Mohammed Sayagh examines the real impact of instruction files on AI-generated pull requests, known as agentic PRs. Analyzing 15,549 PRs from 148 open-source projects, the study compares project performance before and after developers created instruction files for AI coding agents like GitHub Copilot. The results are mixed: while 27.7% of projects saw their merge rate improve by at least 20%, roughly the same proportion (26.35%) actually experienced a decline.
The study evaluated performance across three dimensions: merge rate, code churn (volume of changes), and effort to merge (time and comments). In all dimensions, the introduction of instruction files did not consistently improve outcomes. However, a deeper analysis revealed that projects that succeeded had instruction files that were substantially longer and better organized into multiple sections and subsections.
The authors argue that effective instruction files should be treated as software artifacts, advocating for an "Instructions-as-Code" approach where developers apply software engineering best practices to crafting these guides. The findings underscore that simply adding instructions is not enough—the quality, structure, and maintenance of these files are critical for AI agent performance.
- Analyzed 15,549 agentic PRs from 148 projects to measure instruction file impact
- 27.7% of projects saw merge rate increase ≥20%, while 26.35% saw decreases
- Successful projects used longer, well-structured instruction files with more sections
Why It Matters
For teams using AI coding agents, quality of instruction files can make or break PR success rates.