StreamGuard slashes HPC streaming failures by 6x with <1% overhead

StreamGuard, developed by Hai Duc Nguyen, Bogdan Nicolae, Tekin Bicer, and colleagues at Argonne National Lab, University of Chicago, and French research institutes, targets a fundamental bottleneck in real-time scientific workflows: the producer-consumer streaming pattern. Traditional resilience methods either halt computation or add heavy overhead, which violates strict real-time constraints. StreamGuard introduces two complementary techniques: (i) a dynamic, asynchronous, non-blocking checkpointing mechanism that preserves progress without interrupting the data flow, and (ii) a progress-aware load redistribution strategy that proactively detects stragglers (due to hardware faults, network issues, or resource contention) and rebalances tasks across healthy workers. Together, these mechanisms ensure forward progress even in highly error-prone environments, without requiring manual tuning or application changes.

Experimental results on HPC clusters demonstrate that StreamGuard reduces the impact of failures and performance anomalies by up to 6x compared to baseline approaches, while introducing less than 1% overhead in failure-free scenarios. This is significant for domains like real-time climate modeling, particle physics, and medical imaging, where continuous data ingestion and low-latency output are critical. The framework is designed to be integrated into existing streaming systems (e.g., Apache Flink, Spark Streaming) used in HPC settings. The paper, accepted at ICS'26, also releases code and benchmarks via the arXiv repository, making it immediately usable for researchers and engineers building fault-tolerant real-time pipelines.