PyTorch Fixes cumprod Backward Tests with `rand` over `randn`
Using normal distribution caused 1.6% mismatch in cumulative product gradient tests.
PyTorch has merged a small but critical test fix (PR #188852) that addresses flaky failures in cumulative product (cumprod) backward gradient tests. The issue arose because `torch.randn` (normal distribution) produces values with wide dynamic range, which can cause floating-point associativity differences when computing gradients through the cumulative product operation. This manifested as an assertion error in `test_cumprod_backward_split_scan_reduction_fusion` where 1 out of 64 elements mismatched by up to 0.186 (absolute) and 0.0076 (relative), exceeding the allowed tolerances of 0.01 and 0.002 respectively. The error was intermittent and depended on random seed values.
The fix, authored by eqy and approved by Skylion007, simply replaces `randn` with `rand` (uniform distribution in [0,1)) in the test harness. Uniform values avoid the extreme outliers that amplify associativity-related precision differences in the backward pass of cumulative operations. This change makes the test deterministic and reliable across different hardware and compiler backends (e.g., Inductor). While the fix is minor, it ensures that continuous integration pipelines for PyTorch are not blocked by spurious failures, and developers depending on cumprod (common in attention masks, normalizing flows, and sequence processing) can trust the gradient stability.
- PR #188852 replaces `randn` (normal) with `rand` (uniform) in cumprod backward tests to avoid associativity failures.
- The fix resolves a 1.6% tensor mismatch error (absolute diff 0.186) in the `test_cumprod_backward_split_scan_reduction_fusion` test.
- Approved by PyTorch maintainer Skylion007, authored by eqy using Codex.
Why It Matters
Ensures reliable gradient tests for cumulative product operations, a staple in sequence models and normalizing flows.