AMD's ROCm Update Fixes AI Memory Bug, Speeding Up Training
This fix could make AI models train faster on AMD hardware, saving time and money.
The one_shot_all_reduce Inductor lowering and its registered symm-mem buffer (empty_strided_p2p) now work on ROCm >=10.1, according to the pull request. Because of that, the skip is being changed to be ROCm version dependent. Since CI has been upgraded to ROCm 10.1, these UTs are now enabled in CI.
The PR fixes #168536 and #168537, is supported by @jeffdaily, and was approved by jeffdaily. In the test plan, on 2x gfx942, running HIP_VISIBLE_DEVICES=0,1 pytest test/distributed/test_symmetric_memory.py -v -k LoweringTest passes. The pull request resolved is #198966, co-authored by Claude Opus 4.8 (1M context).
- AMD's ROCm 10.1 update fixes a memory-sharing feature for AI training.
- The fix makes AI training faster and more efficient on AMD chips.
- It increases competition with Nvidia, potentially lowering AI computing costs.
Why It Matters
Faster, cheaper AI training on AMD chips could lead to more affordable AI services for everyone.