Shaker Tool Promised to Catch Random Software Bugs — Study Says No

When software teams build apps, they also write automated 'tests' — little robot checks that confirm the software still works. Sometimes a test passes one minute and fails the next, even though nobody changed the code. Engineers call these 'flaky tests,' and they are a huge time sink: teams hunt bugs that don't exist, delay releases, and stop trusting their own safety net.

A tool called Shaker was built to find them. Instead of just re-running tests, Shaker deliberately stresses the computer — maxing out processor, memory and disk activity — to force hidden randomness into the open. In earlier trials on Java and Android projects, it reportedly caught 95% of flaky tests, compared with 37.5% for plain re-running. That made it look like a near-miracle for software quality.

Researchers at Brazilian universities ran the first independent test of Shaker on Python. They took 137 tests already known to be flaky and ran each one 100 times with Shaker and 100 times with plain re-running, a fair, budget-matched comparison. The result: 37.2% versus 35.8%. Statistically, no difference at all. So the tool, in the configuration that worked for Java, offered Python teams nothing extra.

Why? Two reasons. First, fewer than half of the supposedly flaky tests actually behaved flakily on the new machine — most never failed even once in 100 runs. Second, the flakiness that did appear came mostly from network connections and random numbers, not the concurrency problems Shaker is designed to trigger. The bigger warning is for the research field itself: if you measure a tool against a list of known-bad tests, but a different computer makes those tests look healthy, every tool gets unfairly marked down. Fixing that measurement gap matters more than any single tool.