Risk-Aware Batch Testing for Performance Regression Detection

A team of researchers has published a novel framework that uses AI to dramatically optimize performance regression testing in continuous integration (CI) systems. The core innovation unites two previously separate approaches: machine learning models that predict the risk of a code commit causing a performance regression, and adaptive batching strategies that group commits for testing based on that predicted risk. Using Mozilla Firefox's massive Autoland CI system as a real-world case study, the team created a dataset of confirmed regressions and fine-tuned several transformer models—including CodeBERT, ModernBERT, and variants of Meta's LLaMA-3.1—to score each commit's risk. CodeBERT performed best, achieving a 0.694 ROC-AUC score in identifying risky commits.

These risk scores then drive new batching algorithms like Risk-Aged Priority Batching (RAPB). Instead of testing every commit immediately or using simple first-in-first-out queues, RAPB intelligently prioritizes high-risk commits for faster testing while batching lower-risk ones, creating a more efficient testing pipeline. In simulations across thousands of historical Firefox commits, their best configuration (RAPB with linear aggregation) delivered a Pareto improvement: it simultaneously reduced resource consumption and improved diagnostic speed. Specifically, it cut the total number of test executions by 32.4%, reduced the maximum time-to-culprit (the delay in pinpointing a bad commit) by 26.2%, and lowered mean feedback time by 3.8%, all while maintaining the baseline's mean time-to-culprit. The team estimates this approach could save a project of Firefox's scale approximately $491,000 annually in cloud compute and infrastructure costs. The complete replication package, including datasets and code, has been released to support industry adoption and further research.