FAIR-Compute roadmap: UK HPC allocation measures occupancy, not utilization
Study finds federated HPC schedulers can't track productive value across UK systems
The FAIR-Compute project, a final report to the UK's National Federated Compute Services Flexible Fund, tackles a critical but overlooked problem: how scarce high-performance computing is allocated across federated research infrastructure. Led by Konstantinos Zachariadis and colleagues, the team combined a landscape review of UK and international allocation practice (covering EuroHPC, WLCG, ACCESS, JASMIN, and DiRAC), a mechanism-design model accounting for strategic user reports, and simulations on the public Fresco/Anvil workload trace. The central insight is that scheduler settings are not purely technical — they are policy decisions that determine who runs, when, and under what evidence.
Three findings recur across all evidence strands. First, allocation records track occupancy (resources reserved) rather than utilization (useful work completed), so UKRI and DSIT cannot answer their core question: whether research resources deliver productive value. Second, simple, transparent scheduling heuristics perform nearly as well as an offline full-information benchmark once tuned, suggesting near-term gains come from instrumenting existing schedulers, not replacing them. Third, federation behaves like uncoordinated routing: it can minimize average delay while quietly concentrating load on receiving systems, mirroring Braess's paradox in road traffic where individual route choices worsen network outcomes. The roadmap urges shifting to utilization-based allocation metrics and actively managing federation to avoid systemic inefficiency.
- FAIR-Compute analyzed allocation across EuroHPC, WLCG, ACCESS, JASMIN, and DiRAC using mechanism design and Fresco/Anvil workload traces
- Current records measure occupancy, not utilization, leaving UKRI/DSIT unable to assess if compute delivers productive value
- Tuned simple heuristics approach offline optimal benchmarks, while unmanaged federation risks Braess's paradox load concentration
Why It Matters
For UK research, utilization-based HPC allocation could redirect compute budgets from idle reservations to productive scientific work.