AI Agents Show Distinct Code Mutation Patterns in Performance PRs

Researchers from the University of Sunderland and Shiraz University have cracked open the black box of AI coding agents by analyzing what they actually change in performance-improving pull requests. Their paper, submitted to arXiv, reveals that fewer than 1% of the 33,596 agent-generated PRs in the AIDev-pop dataset target performance, making each case a rare window into otherwise opaque agent behavior. By classifying 1,254 diff hunks from 216 PRs across five agent systems, the team applied an 18-category syntactic mutation taxonomy using a dual-LLM intersection pipeline.

The findings are eye-opening: name modification accounts for 37% of mutations, object creation 26.4%, and type change 22.7% — a profile markedly different from prior genetic improvement corpora where no change made up 84%. Each agent system commits to a distinctive mutation vocabulary, and each performance strategy activates a largely disjoint subset of categories. This means that knowing which agent and which strategy is in play can narrow the effective operator space for search-based software engineering, potentially making tools like genetic improvement more efficient and targeted.