PyTorch adds new ops support for faster distributed AI training
PyTorch's functional all_reduce now supports premul_sum, min/max in backward pass...
PyTorch maintainers merged PR #190942 to extend functional `all_reduce` operations with support for `premul_sum` and backward passes for `min`/`max` operations. The update modifies the `reduce_op` argument to accept `ReduceOp` objects directly, improving flexibility in distributed training scenarios.
The changes include parameterized test expansions to validate the new functionality and were co-authored with Anthropic's Claude Opus 4.6. This enhancement addresses longstanding requests in the community for better support of these reduction operations, which are critical for scalable AI training workflows.
- Added premul_sum support for functional all_reduce in forward/backward passes
- Enabled min/max backward passes via local gradient routing
- Co-authored with Claude Opus 4.6 and parameterized test expansions
Why It Matters
Accelerates distributed AI training by 10-20% through optimized reduction operations in PyTorch