Cross-Project Flakiness: A Case Study of the OpenStack Ecosystem
Flaky tests are silently wasting millions in compute and developer time across ecosystems.
A new study analyzing 649 OpenStack projects reveals that cross-project flaky tests—where a test fails non-deterministically across multiple projects—affects 55% of the ecosystem. Researchers identified 1,535 such tests, which significantly increase code review time and computational costs. Surprisingly, 70% of these were unit tests, debunking the belief they are isolated. Primary causes include race conditions, inconsistent CI configurations, and dependency mismatches, highlighting a systemic reliability crisis in large-scale software development.
Why It Matters
This exposes a hidden, costly inefficiency that undermines trust in automated testing and CI/CD pipelines for major software platforms.