Llama.cpp v.b9685 adds SYCL dev2dev memcpy for GPU direct transfers
New release enables direct GPU-to-GPU memory copies via SYCL API, improving multi-GPU inference.
The open-source llama.cpp project, led by ggml-org, has released version b9685 with a significant update to its SYCL backend. The headline feature is the addition of dev2dev memcpy functionality using the SYCL API, which allows direct memory copies between GPUs without staging through the host. This is essential for efficient multi-GPU inference, as it reduces latency and bandwidth overhead. The update also refines p2p (peer-to-peer) communication detection methods and moves the GGML_SYCL_DEV2DEV_MEMCPY flag into a runtime configuration table, giving users more control over memory transfer strategies.
This release continues llama.cpp's tradition of broad platform support. Pre-built binaries and build configurations are provided for macOS (Apple Silicon with and without KleidiAI, Intel x64, iOS XCFramework), Linux (Ubuntu x64/arm64/s390x CPU, Vulkan, ROCm 7.2, OpenVINO, SYCL FP32/FP16), Windows (x64/arm64 CPU, CUDA 12/13, Vulkan, OpenVINO, SYCL, HIP), and Android arm64 CPU. OpenEuler builds are disabled in this release. The SYCL enhancements are particularly valuable for users with Intel or AMD GPUs that support SYCL, enabling them to run large language models like Llama, Mistral, and Gemma across multiple GPUs with improved performance.
- Adds dev2dev memcpy via SYCL API for direct GPU-to-GPU memory transfers
- Fixes p2p communication detection and moves device-to-device memory to runtime config
- Supports macOS, Linux, Windows, Android across CPU, CUDA, Vulkan, ROCm, OpenVINO, SYCL, HIP backends
Why It Matters
Enables faster multi-GPU local inference for large models, reducing host bottlenecks and improving throughput.