AMD Research paper cracks NUMA GPU memory sharing to speed up LLM inference

As LLMs grow, GPU makers are turning to multi-partition designs with non-uniform memory access (NUMA) characteristics, where compute blocks and memory sit in physically separate regions. This can create contention and poor data locality, dragging down kernel latency. In a new arXiv paper (2607.28824), Donghyeon Joo and colleagues from AMD Research and the University of California analyze performance-critical LLM kernels—weight projection, mixture-of-experts, and attention variants—from state-of-the-art serving engines. They introduce a memory trace analysis methodology that derives workgroup-level data access and sharing behavior, then validate it using a cycle-level simulator to measure the impact on latency.

The authors categorize LLM kernel operands into three inter-workgroup sharing patterns: global, partial, and private. Each category demands a different optimization strategy. Private operands benefit from simple per-workgroup data pinning to keep memory local, while global and partial sharing require more sophisticated subgroup-aware co-scheduling to minimize cross-partition traffic. The paper shows that naive kernel implementations often ignore these NUMA effects, leading to suboptimal performance on multi-partition GPUs. The findings point to the need for placement-aware kernel programming and smarter architectural support for work and data locality—a practical roadmap for GPU engineers aiming to squeeze more throughput from next-gen hardware.