New study predicts AI coding agent failures with 86% accuracy

A new empirical study from researchers Ebtesam Al-Haque and Brittany Johnson tackles a growing problem in AI: agentic coding systems are saturating benchmarks, yet nobody knows what actually makes one issue resolution task harder than another. Using CoderForge-Preview, the largest open dataset of coding agent trajectories to date, they extracted features across task patches, repositories, and prompts, then evaluated their predictive power against task outcomes using ensemble methods, SHAP attribution, and effect size analysis.

The results are striking: task difficulty is highly predictable from static properties alone, achieving an AUC of 0.863. The dominant drivers are patch fragmentation and repository scale — meaning how scattered the required changes are and how large the codebase is — while prompt linguistic features only become significant for mid-complexity tasks. This layered structure of difficulty suggests that benchmark scores can be misleading without controlling for these factors. The authors propose a measurement framework that allows static, pre-hoc difficulty estimation, laying the groundwork for difficulty-controlled benchmark construction. For teams evaluating coding agents, this means you can now predict which tasks will trip up your AI before you even run them, and build fairer tests that actually measure capability rather than luck.