PyTorch fixes FlexAttention backward pass for score_mod without gradients
No more InductorError when score_mod ignores the raw score input.
FlexAttention in PyTorch builds a joint graph for score_mod so the backward pass can compute the gradient of modified scores w.r.t. raw attention scores. When score_mod returns a value completely independent of score (e.g., a mask based only on indices), AOTAutograd correctly reports no gradient for the score input. However, the FlexAttention lowering did not handle that None result – Inductor still expected a score-gradient subgraph output for the dq/dk matmuls, causing backward to fail during lowering/kernel generation.
The fix addresses this by materializing a zero score gradient when the joint graph returns None for the score input. Constant joint graphs can lower that zero as a scalar/rank-1 Triton value, so the backward template now broadcasts those low-rank grad_scores values to the score tile before the matmuls. Rank-2 gradients from normal differentiable score_mod paths skip the extra broadcast add. The patch also fixes the higher-order op contract by ensuring the joint graph always provides a score-gradient value to the backward lowering. Benchmarks on CUDA fp16 (B=1, H=1, S=512, D=64, score_mod=score*1.1) show no clear regression: patched median 0.5404 ms vs baseline 0.5343 ms, within run-to-run noise.
- Bug: score_mod returning a value independent of score (e.g., a mask) caused InductorError during backward because the gradient was None.
- Fix: materializes a zero gradient in create_fw_bw_graph and broadcasts low-rank values in the backward template to avoid extra ops.
- No performance regression: median backward time 0.5404 ms (patched) vs 0.5343 ms (baseline) on a typical CUDA benchmark.
Why It Matters
Enables robust FlexAttention usage with arbitrary score_mod functions, removing a crash barrier for attention pattern research.