Meet ARGUS: The Always-On GPU Monitor That Slashes Diagnosis Overhead to Under 2% — Even on 10,000+ GPU Clusters

Large-scale LLM training demands always-on observability, but existing solutions are either too coarse for root cause analysis or too expensive (5-30% overhead) for production use. ARGUS solves this by decomposing the training call hierarchy into CPU call stacks, framework semantics, and GPU kernel execution—all collected with a combined overhead under 2%. To manage trace volume, it builds a unified pipeline that compresses raw kernel events by approximately 3,700x, reducing data from 10 MB to just 2.7 KB per rank per step. This makes always-on, fine-grained tracing practical for the first time in 10,000+ GPU production clusters.

Deployed for over six months on a 10,000+ GPU cluster, ARGUS provides a progressive diagnosis framework that automatically isolates anomalous windows, straggler ranks, and degraded kernels through three analysis levels: iteration-time, phase-level, and kernel-level. Case studies demonstrate its effectiveness against diverse issues including compute stragglers, link degradation, pipeline-bubble amplification, FlashAttention JIT stalls, and compute stragglers masked by communication symptoms. The system supports continuous fail-slow detection and ongoing performance optimization, offering a new standard for observability in distributed AI training.