arXiv paper: flawed weak scalability tests mislead visualization algorithm benchmarks

Empirical scalability tests are the backbone of high-performance visualization research, but they're often built on a shaky assumption: that increasing problem size by scaling input data actually increases algorithmic workload proportionally. In their new arXiv paper (arXiv:2608.08166), Marvin Petersen, Jonas Lukasczyk, and Christoph Garth systematically dismantle this assumption. They apply multiple common data scaling methods—such as replication, resampling, and synthetic generation—across different visualization algorithms and datasets. The results confirm that weak scalability (where problem size and computing resources grow together) is heavily distorted when an algorithm's workload depends on factors like input data complexity or output size. A larger input file doesn't necessarily mean more work for the algorithm if the underlying structure stays simple.

To address this, the authors introduce a method that mitigates workload inconsistencies for shared-memory settings. Their approach goes beyond naive input-size scaling by accounting for how each algorithm actually consumes computational resources, yielding more honest weak scalability curves. The paper—19 pages with 15 figures—is a methodological critique as much as a technical contribution, calling for standardized practices in how scalable visualization research is evaluated and reported. For researchers working on petascale or exascale visualization, this is a crucial reminder that benchmark numbers can lie, and it offers a practical path toward benchmarks that reflect real-world scaling behavior.