Developer Tools

PyTorch #189486 adds worker phase reporting to sidecar watchdog for better debug

Distinguish stuck autotune-cache from stuck Triton compile with no extra IPC overhead.

Deep Dive

PyTorch PR #189486 introduces phase reporting for the sidecar watchdog that monitors compile workers. Previously, when a worker got stuck, the watchdog could only report total elapsed time, making it impossible to tell whether the worker was hung in an autotune-cache lookup or during Triton compilation. This change adds a shared multiprocessing.Array allocated in the sidecar before it forks its worker pool (fork-only; spawn pools degrade to duration-only). Each forked worker inherits the buffer and writes its own slot — containing job ID, phase enum value, phase-start time, and PID — with no locking and no extra IPC on the compile hot path. The watchdog thread reads all slots each tick and folds the phase info into the STATUS report it already sends the parent process. The parent emits this in the compile_worker_status TLParse artifact as phase, phase_elapsed_s, and worker_pid (or phase "queued" if the job hasn't been picked up yet).

The implementation touches several files: compile_worker/watchdog.py (the shared buffer and phase API), subproc_pool.py (allocate before fork, stamp job/phase in do_job, read in the watchdog loop), and two checkpoint sites — QUERYING_CACHE around the autotune-cache read and COMPILING before Triton precompile. The Phase enum is append-only since its values live in the buffer. Tests include a new test that asserts a worker reporting QUERYING_CACHE surfaces phase=querying_cache with elapsed time and PID in the STATUS report. The shared-memory heartbeat was also verified standalone with real os.fork. Authored with Claude.

Key Points
  • Shared multiprocessing.Array allocated before fork (fork-only) stores worker phase, job ID, start time, and PID without locking.
  • Two new phase checkpoints: QUERYING_CACHE around autotune-cache reads and COMPILING before Triton precompile.
  • Parent emits phase info in compile_worker_status artifact as phase, phase_elapsed_s, and worker_pid.

Why It Matters

Faster debugging of wedged PyTorch compile workers by pinpointing exactly which phase is stuck.

📬 Get the top 10 AI stories daily