PyTorch patches native_layer_norm to fix CUDA mixed-dtype silent errors
Layer norm with bf16 input and fp32 parameters now fails correctly instead of silently corrupting output.
PyTorch contributor jansel landed a fix (PR #185693) for a subtle inconsistency in the `native_layer_norm` decomposition. The issue: PyTorch's `aten.native_layer_norm` CUDA kernel requires that non-empty weight and bias parameters have the same dtype as the input tensor. However, the Python reference decomposition (used in tracing, fake tensors, and `aot_eager_decomp_partition`) did not enforce this restriction. It would silently promote mixed dtypes, cast the output back, and accept programs that the actual CUDA kernel would reject with an error. This meant a workflow using `aot_eager_decomp_partition` could generate a graph that would fail at runtime on GPU.
The fix adds a CUDA-specific check in the shared decomposition path (not a backend-specific hook) to validate that weight and bias dtypes match the input dtype when the layer norm has non-empty rows. For empty-row inputs (where no kernel launch occurs), the native behavior is preserved. The patch also includes comprehensive tests: `CUDA repros` tests and CPU decomposition tests ensure correctness. Benchmarking on a bf16 input of shape (32, 128) shows no performance hit — median time actually slightly improved from 79.42 µs to 77.98 µs. This PR, approved by ColinPeppler, resolves issue #151478 and ensures decomposition fidelity matches eager CUDA execution.
- Mixed-dtype weight/bias (e.g., bf16 input + fp32 params) now triggers a proper error in decomposition, matching CUDA kernel behavior.
- Fix applies to shared decomposition path used by fake tensors and aot_eager_decomp_partition, preventing invalid graphs from being accepted.
- Benchmarks confirm no regression: median runtime improved from 79.42 µs to 77.98 µs on CUDA bf16 (32, 128) shape.
Why It Matters
Eliminates silent correctness bugs in PyTorch's layer norm decomposition, ensuring graph capture mirrors eager execution faithfully.