EnsembleLauncher orchestrates 8M tasks 4x faster on exascale supercomputers
New decentralized orchestrator handles 8 million tasks, 4x faster than current tools
As scientific computing moves toward coupled simulation-AI workflows, orchestrating millions of heterogeneous tasks on leadership-class systems creates severe bottlenecks. Traditional system-level schedulers are configured for limited throughput, and workflow tools suffer from rigid control-plane topologies and static scheduling heuristics. The resulting extreme ensemble sizes and task variability hinder scalability and resource utilization, especially in active learning pipelines.
To address these challenges, researchers from Argonne National Laboratory developed EnsembleLauncher, a recursively hierarchical workflow orchestrator with a fully decentralized control plane and a programmable scheduling policy interface. Tested on the Aurora supercomputer, EnsembleLauncher scaled to the entire machine, handling up to eight million serial tasks and achieving a 4x performance improvement over state-of-the-art tools. Its programmable scheduling interface allows users to adapt policies dynamically for high-variance ensembles, significantly improving resource utilization in modern coupled simulation-AI workflows.
- Scales to 8 million serial tasks on the Aurora supercomputer
- Outperforms state-of-the-art tools by over 4x in task orchestration throughput
- Features a programmable scheduling policy interface for high-variance ensembles and active learning
Why It Matters
Enables efficient orchestration of massive simulation-AI workflows on exascale systems, accelerating scientific discovery.