Mozilla's ensemble CPD method improves performance anomaly F1-score by 11%
Mozilla's Perfherder currently has 12.5% false positives and misses 6.8% regressions.
At Mozilla, performance regression detection is handled by Perfherder, which uses a Student's T-test to flag anomalies across hundreds of daily code changes. However, an analysis of one year of data revealed that 12.5% of alert groups were false positives and 6.8% contained regressions missed by the system — highlighting a clear need for improvement.
To address this, Mozilla researchers conducted an empirical study of 25 change-point detection (CPD) methods and 15 ensemble approaches, using a manually annotated benchmark of 174 performance time series created by 11 Mozilla performance engineers. While individual offline and hybrid CPD methods improved recall, they did so at the cost of precision. Ensemble voting strategies offered a better trade-off, yielding an 11% improvement in F1-score. The team validated the results and integrated the best-performing methods into Mozilla's production performance engineering system.
- Mozilla's Perfherder generates 12.5% false positive alerts and misses 6.8% of regressions with its current T-test approach.
- Researchers tested 25 CPD methods and 15 ensembles on 174 manually annotated time series by 11 engineers.
- Ensemble voting improved F1-score by 11% while balancing precision and recall in production integration.
Why It Matters
Better anomaly detection in CI pipelines means fewer false alarms and faster catching of real regressions for developers.