Developer Tools

PyTorch fixes Inductor backward crash when multiple compile frames run

⚑A subtle registry clobber bug caused CUDA backward failures in training pipelines.

Deep Dive

PyTorch has merged a critical fix for its TorchInductor compiler, addressing a concurrency bug that could silently corrupt training pipelines using multiple `torch.compile` frames. The issue: when multiple compiled functions share a global registry of external objects (like CUDA streams and events), later frames could overwrite weak references used by earlier backward passes. This caused the backward pass to throw "Index not registered in index_to_external_object_weakref" when trying to retrieve its required resources.

The solution, proposed by mlazos and co-authored with Claude, introduces a snapshot mechanism. After the forward pass completes, the registry is copied by dereferencing weakrefs into strong references stored on the autograd node (`ctx`). During the backward pass, the snapshot is restored by writing weakrefs directly back into the global registry, avoiding the memory overhead of `set_external_object_by_index`. This ensures backward has the correct objects, and because the strong refs are tied to the autograd node’s lifetime, they are freed once backward completes or the gradient function is dropped. The snapshot logic is shared via a module-level helper called from both finalize and `_codegen_finalize`.

Key Points
  • Multiple `torch.compile` frames could clobber the global external object registry, causing backward failures on CUDA streams/events.
  • Fix snapshots the registry after forward by converting weakrefs to strong refs stored on the autograd node.
  • Snapshot restored in backward using direct weakref writes to avoid memory growth; strong refs freed when backward completes.

Why It Matters

Fixes a subtle, hard-to-debug training crash that affected multi-frame compiled pipelines β€” critical for production PyTorch users.

πŸ“¬ Get the top 10 AI stories daily