AI Can Now Predict Which App Tests Fail for No Reason
Ghost test failures waste thousands of developer hours and delay the apps you use.
Deep Dive
Researchers trained five classifiers to spot "flaky tests" — software checks that produce non-deterministic outcomes without any code change, eroding CI confidence and delaying deliveries. Studying 91 flaky and 22,349 stable tests from 15 open-source Swift projects, the best model — Random Forest — flagged flakiness with 92% precision and an F1 of 0.86. Telltale words like "async", "await", "wait" and "timeout" reveal the trouble spots.
Key Points
- Flaky tests are automated checks that randomly pass or fail without any code change, making developers distrust their own testing and waste hours re-running them.
- Researchers trained AI models on 22,349 stable and 91 flaky Swift tests — the best one flagged unreliable tests with 92% precision and 95% accuracy.
- Certain words are red flags (async, wait, timeout, expectation) while plain assertions like XCTAssertEqual signal safety — but the method misses problems hidden outside the test body.
Why It Matters
Fewer ghost test failures means faster fixes and more reliable app updates on the phone in your pocket.