Combining Software Estimates Often Backfires, Study Finds
A new study shows that blending estimates can sometimes make predictions worse, not better.
When you need to predict how long a software project will take, you might think that combining several estimates would give a better answer. But a new study suggests that's not always true. Researchers tested this idea on four public datasets of software projects, using three common prediction methods and, in one case, a human expert's estimate.
The results were mixed. In the dataset that included a human estimate, that human was so accurate that combining it with the other methods made predictions much worse—like adding water to a strong cup of coffee. In the other three datasets, combining did improve accuracy slightly, by 1.3% to 8.6%, but these gains were so small they could have been due to chance.
The lesson: combining estimates is not a magic bullet. If you have one very reliable source, don't dilute it. If you have several equally good sources, combining might help a little, but don't expect miracles. The study also found that more sophisticated combination methods (like those that account for how errors are correlated) performed better than simple averaging, but the differences were modest.
For project managers and teams, this means you should be cautious about blindly averaging estimates. Instead, evaluate the quality of each estimate and consider whether combining is worth the effort. In many cases, the simplest approach—using the best single estimate—may be just as good.
- Combining multiple estimates can sometimes make predictions worse, especially if one estimate is much better than the others.
- In three out of four datasets, combining gave small improvements (1.3% to 8.6%), but these were not statistically significant.
- Simple averaging was often outperformed by more complex methods that account for how errors are correlated.
Why It Matters
Blindly averaging estimates can waste time and money; better to use the most reliable single estimate.