[P] TraceML: wrap your PyTorch training step in single context manager and see what’s slowing training live
Wrap your training step in one line to get live GPU memory, DDP rank imbalance, and dataloader timing data.
TraceML has released a new open-source debugging tool designed to give PyTorch developers instant visibility into what's slowing down their model training. The tool's core innovation is a single context manager—`with trace_step(model):`—that users wrap around their training step. Once active, it provides a live dashboard that surfaces critical performance metrics while the training job runs, eliminating the need for post-hoc log analysis or manual instrumentation.
The dashboard displays key metrics including dataloader fetch time, the breakdown of forward pass, backward pass, and optimizer step timing, and GPU memory usage. Crucially for multi-GPU setups, it visualizes rank imbalance in single-node Distributed Data Parallel (DDP) training, highlighting if one GPU is a straggler causing bottlenecks. At the end of a run, it generates a compact summary pinpointing the slowest rank and step. The tool currently supports single-GPU, single-node multi-GPU DDP, and integrates with popular frameworks via Hugging Face Trainer and PyTorch Lightning callbacks.
The primary goal is to answer the common, frustrating question: "Why is this training run slower than it should be?" By making runtime inefficiencies immediately visible, TraceML helps engineers and researchers quickly diagnose issues like slow data pipelines, memory spikes, or synchronization delays in distributed training. The company is actively seeking user feedback on the GitHub repo to refine the signals it captures.
- Adds runtime visibility with a single line of code: `with trace_step(model):`.
- Live dashboard shows dataloader timing, GPU memory, and DDP rank imbalance to find stragglers.
- Generates an end-of-run summary highlighting the slowest step and rank for quick debugging.
Why It Matters
Saves engineers hours of manual profiling by instantly revealing the root cause of slow PyTorch training jobs.