Hybrid workflow method boosts HL-LHC data processing by 3.8x

A new study from researchers Alan Malta Rodrigues and Douglas Thain tackles a critical bottleneck in extreme-scale data processing: workflow composition. Focusing on the High-Luminosity Large Hadron Collider (HL-LHC), the paper presents a simulation framework that optimizes how tasks are grouped within a Directed Acyclic Graph (DAG). The key insight is that hybrid composition strategies—dynamically balancing task independence with execution grouping—can dramatically improve performance. The authors explored a high-dimensional parameter space and found that their approach yields up to 3.8x throughput increase and a 14.9x reduction in network overhead compared to static methods.

Beyond raw speed, the framework introduces a multi-metric objective function that lets system architects navigate trade-offs between throughput, I/O cost, and CPU efficiency. This enables policy-driven optimization, making it possible to automatically synthesize workflows tailored to specific infrastructure constraints. The study was validated against real HL-LHC workloads and all artifacts are publicly available. Accepted at IEEE eScience 2026, the work provides a rigorous foundation for automated workflow synthesis in distributed systems, promising to accelerate scientific discovery in particle physics and other data-intensive fields.