Avoid GPU stalling while profiling PyTorch training with CUDA events

Profiling PyTorch training introduces a classic observer effect: measuring performance can alter it. The common practice of calling torch.cuda.synchronize() gives clean timing boundaries but forces GPU synchronization, stalling the asynchronous CUDA pipeline and distorting actual run behavior. traceml-ai's technical note proposes using CUDA events as an alternative—marking start and end points asynchronously and reading timestamps later, without interrupting the hot path. This yields more accurate profiling for real-world training workloads.

This method is not meant to replace full-featured profilers like PyTorch Profiler or NVIDIA Nsight, but acts as a lightweight first pass for quick diagnostics. It is part of an open-source PyTorch training diagnostics tool, offering developers a low-overhead way to spot bottlenecks before diving into operator-level analysis. The technique is especially valuable for distributed or large-scale training where even minor synchronization can skew results.