GPUAlert: 0-instrumentation tool catches 40% GPU training-job failures

GPU training jobs fail roughly 40% of the time on large production clusters, yet operators often learn of failures only hours later via a single status line with no cause or logs. Existing solutions require editing training scripts or maintaining cloud connections. GPUAlert, a new open-source tool from researchers Parv Agarwal and Asif Ekbal, solves this by acting as a zero-instrumentation process-boundary monitor. It wraps any training command without changes and, upon completion, emails a structured notification containing a classified failure cause, durable logs, and output artifacts. The tool is built on three reliability primitives: pre-launch log guarantee (establishes durable storage before the child process can crash), notifier isolation (makes the wrapper's exit code a pure function of the child's status regardless of email success), and a non-silent artifact budget (bounds attachment size without dropping output).

Alongside the tool, the authors release a labeled corpus of 474 GPU training logs spanning 15 failure classes and a reproducible evaluation harness. In tests across twelve hardware-reproduced classes, GPUAlert's ordered-rule classifier achieves 0.997 macro-F1, dramatically outperforming unordered keyword matching (0.830) and exit-code inspection (0.133). Overhead is a constant ~3ms per job, and the pre-launch guarantee preserves logs even when shell redirects yield nothing. Crucially, the wrapper returns the child's exit code unchanged even when the SMTP relay is unreachable. For ML engineers training large models, GPUAlert automates failure diagnosis with zero code changes, potentially saving hours of debugging per incident.