New study predicts AI vs human PR acceptance with 95% F1 accuracy

A new paper by Kartik Ghanshyambhai Pansuriya, Ehsan Ghorbani, Deepak Singh, and Eman Abdullah AlOmar investigates whether machine learning can predict pull request (PR) acceptance and review effort from submission-time data alone. Using the AIDev dataset, which contains both human- and AI-agent-authored PRs, the researchers built a leakage-aware prediction pipeline that excludes any post-submission signals like reviewer comments or CI status. The feature set includes PR text characteristics, metadata, repository context, temporal signals, and lightweight diff statistics. They evaluated Logistic Regression, Random Forests, Gradient Boosting, Extra Trees, and Multi-Layer Perceptrons across pooled, human-only, agent-only, and balanced contributor views.

Results show that acceptance prediction is highly feasible: tree-based models achieve F1 scores above 0.95, with textual clarity and metadata being the most influential predictors. This means maintainers can triage incoming PRs early, flagging likely rejects before any human effort is spent. However, predicting review effort (measured by comment count and time-to-merge) proved far more difficult—submission-time features only modestly explain these outcomes. The authors conclude that while early PR models can support triage and reviewer prioritization, they should remain advisory tools rather than automated gatekeepers, as team-specific workflows and reviewer availability dominate effort estimation.