HPC Containers for EBRAINS: Towards Portable Cross-Domain Software Environment

A research team from the EBRAINS Research Infrastructure has published a breakthrough paper demonstrating how to create portable HPC container images that maintain bare-metal performance across diverse computing environments. Using Apptainer container technology with a hybrid PMIx-based strategy, their approach dynamically leverages host-level specialized hardware including network interfaces and GPUs on production HPC clusters like Karolina and Jureca-DC. This solves the long-standing problem of site-specific dependencies that typically hinder deployment of complex scientific workflows.

The system targets the EBRAINS Software Distribution (ESD), a Spack-based ecosystem containing approximately 80 top-level packages with 800 dependencies. Crucially, the researchers verified performance using communication microbenchmarks (OSU and NCCL) alongside neuroscience applications, confirming the containers correctly leverage site-installed drivers and hardware. Their verification methodology extends beyond runtime measurements to analyze underlying debug logs, actively detecting misconfigurations like suboptimal transport pathways.

This work demonstrates a reproducible methodology for decoupling software environments from underlying infrastructure, paving the way for automated pipelines that ensure optimized execution across varied HPC architectures. The approach enables researchers to package complex, distributed scientific workflows once and deploy them anywhere without sacrificing the performance needed for computationally intensive neuroscience simulations and analyses.