StageFrontier detects GPU training stalls with 99% less data

When a distributed training job slows down, finding the root cause is notoriously hard because synchronization masks which GPU actually stalled. Standard dashboards using per-stage averages or maxima double-count delays or bury slow ranks, while full profilers like PyTorch Profiler or Nsight Systems produce gigabytes of traces too heavy to leave on. StageFrontier, introduced by researchers Boram Yoon, Wei Chen, and Ville Kallioniemi, closes this gap with a lightweight, always-on signal.

StageFrontier works by having each rank report only a short ordered vector of coarse stage durations—data loading, forward pass, backward pass—measured with CPU wall-clock time (no synchronized clocks or kernel tracing). At each stage boundary, it takes the cumulative time of whichever rank is furthest along; the increments of this frontier form an exact, additive accounting of the step's exposed time. This directly points operators to the stage and rank where the group-visible delay first appeared, telling them where to aim a heavy profiler rather than guessing. In a PyTorch implementation across 128 ranks using Gloo and NCCL, StageFrontier added less than 0.2% throughput overhead, placed injected faults among its top two suspects in all test cases, and recovered the same top-stage routing as full profilers—from a 0.11 MB summary instead of a 15.81 GB trace.