Synthetic benchmarks mislead parallel SSSP algorithms, study finds
Real-world graphs have heavy-tailed weights, not uniform ones —
A new paper from researchers Marco D'Antonio, Thai Son Mai, and Hans Vandierendonck at Queen's University Belfast reveals a critical flaw in how parallel Single-Source Shortest Path (SSSP) algorithms are benchmarked. Most evaluations rely on synthetic graphs with uniformly distributed edge weights, but real-world graphs—from road networks to social media—exhibit heavy-tailed, skewed weight distributions. The team statistically characterized the weight distributions of 17 real-world graphs and contrasted them with six synthetic distributions commonly used in the literature.
Through a comprehensive evaluation of seven state-of-the-art parallel SSSP algorithms, the study demonstrates severe sensitivity to edge weights. Using synthetic uniform weights not only changes optimal parameter configurations but can completely invert the performance hierarchy of algorithms. This means that many published results may not translate to real-world performance, and algorithm designers risk optimizing for artificial benchmarks rather than practical use cases. The findings call for a rethinking of benchmarking practices and more realistic evaluation standards in distributed computing.
- 17 real-world graphs from multiple domains analyzed; all show heavy-tailed weight distributions unlike synthetic uniform weights.
- Seven state-of-the-art parallel SSSP algorithms exhibited severe performance sensitivity to edge weight distribution.
- Synthetic uniform weights altered optimal parameter tuning and could invert the performance ranking of algorithms.
Why It Matters
This challenges decades of SSSP benchmarking and forces the community to adopt real-world weight distributions for fair evaluations.