Developer Tools

PyTorch fixes FakeTensorUpdater for flex attention backward pass

A fix for stale metadata in flex attention backward when wrapped by control_deps.

Deep Dive

A recent pull request to PyTorch (PR #186410) addresses a subtle bug in the inductor compiler's FakeTensorUpdater when handling the backward pass of flex attention. The issue arises when overlap scheduling wraps the flex_attention_backward operation inside a control_deps subgraph. This wrapper creates a new one-node subgraph while the real backward subgraphs (fw, joint, and mask) become operands. The previous control_deps handling only tracked the wrapper subgraph itself, causing FakeTensorUpdater to miss the nested flex subgraphs and leave their placeholder metadata unchanged.

The fix operates on two levels. First, it extends _extract_subgraphs_and_args with a specific mapping for flex_attention_backward, which has a different subgraph signature than forward flex attention. Forward uses score and mask subgraphs with separate optional buffer lists; backward uses fw and joint subgraphs that consume score-mod buffers from args[12], and a mask subgraph from args[13]. This mapping translates HOP operands back to the correct placeholder values for each subgraph. Second, it teaches the control_deps pass to look through its wrapper when wrapper operands contain tracked subgraphs. Unwrapping maps wrapper placeholders back to the actual control_deps operands and reuses the wrapped node's existing HOP extraction logic, avoiding duplication of flex-specific logic.

The regression test uses make_fx to trace the flex backward FX graph, then mutates only the buffer operands that FakeTensorUpdater is expected to repair. The distributed test exercises the end-to-end overlap_scheduling plus insert_overlap_deps path. This fix ensures correct fake tensor propagation, which is critical for accurate shape and memory optimization during compilation of attention modules in large-scale training.

Key Points
  • Flex attention backward has a different subgraph signature with fw, joint, and mask subgraphs consuming buffers from specific argument positions.
  • A control_deps wrapper could cause FakeTensorUpdater to miss nested subgraphs, leaving stale metadata.
  • Fix adds backward mapping in _extract_subgraphs_and_args and teaches control_deps to unwrap to propagate updates correctly.

Why It Matters

Ensures accurate fake tensor propagation in PyTorch's inductor for flex attention, vital for training large models.

📬 Get the top 10 AI stories daily