FAU's ClusterBench exposes 5% performance variance in identical HPC nodes

Data centers have long validated individual nodes with benchmarks, but they lack tooling that validates an entire installation at once. ClusterBench, a new framework from researchers at Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), solves this with cluster-aware scheduling: it dispatches identical benchmarks to every node in a single submission. The bundled benchmark collection targets each key component—CPU, GPU, memory, interconnect, and I/O—so operators can stress-test a full cluster at acceptance and then continuously throughout its lifetime.

Because measurements are repeated over months and years, ClusterBench builds a rich dataset across both space and time. Comparing current runs against earlier baselines automatically flags performance regressions introduced by software changes, such as kernel updates or new library versions. On the NHR@FAU clusters Helma, Alex, and Fritz, the team found that variation within a single component stays within 1%, but variation across nodes with identical specifications reaches 5%—a surprisingly large gap for supposedly homogeneous hardware. By correlating performance with power draw, frequency, and temperature, ClusterBench also revealed that air-cooled and liquid-cooled nodes exhibit different performance-to-power relationships. This makes the framework not just a regression detector but also a research platform for studying hardware variability at scale.