When GPUs Fail Quietly: Observability-Aware Early Warning Beyond Numeric Telemetry

A team of researchers from GWDG (Gesellschaft für wissenschaftliche Datenverarbeitung Göttingen) and the University of Hamburg has published a significant paper on arXiv titled 'When GPUs Fail Quietly: Observability-Aware Early Warning Beyond Numeric Telemetry.' The research tackles a critical problem in modern AI and HPC clusters: a class of GPU failures that occur abruptly with little to no warning in standard performance metrics. These 'detachment-class' failures involve a GPU becoming unavailable at the driver or interconnect level, and the primary observable signal is the structural collapse of the monitoring pipeline itself—metrics disappear, scrape latencies spike, and time-series data develops gaps.

The proposed solution is a novel early-warning framework that performs joint modeling of two signal types. First, it analyzes traditional numeric telemetry like utilization-aware thermal drift. Second, and crucially, it monitors the health of the monitoring infrastructure itself for indicators of degradation. Evaluated on production telemetry from GPU nodes at GWDG, the framework demonstrated that detachment failures exhibit minimal numeric precursors. By incorporating structural signals, the joint model achieved increased early-warning lead time compared to detection methods that rely solely on GPU performance data. The team has also made the dataset used in the study publicly available, providing a valuable resource for further research into AI infrastructure reliability.