Developer Tools

llama.cpp b9939 adds Metal depthwise convolution for faster AI inference on Apple Silicon

New release brings 2D depthwise convolution to Metal, unlocking performance gains for vision models on Macs.

Deep Dive

The b9939 release of llama.cpp, the popular C/C++ implementation of LLaMA and other large language models, introduces native Metal support for depthwise 2D convolution (CONV_2D_DW). Depthwise convolution is a key operation in many computer vision models (e.g., MobileNet, EfficientNet) that reduces computational cost by applying filters per input channel. This patch adds Metal GPU kernels specifically designed for Apple Silicon (arm64), including 3D dispatch for efficient memory access and channel-tiled kernels to handle non-contiguous tensor layouts. The work was co-authored by Georgi Gerganov and includes F16 (half-precision float) support on CPU, along with comprehensive performance tests.

This update marks a significant step in making llama.cpp a viable inference engine for vision tasks on Apple devices. Prior to this, depthwise convolution was either missing or relied on slower generic implementations. By leveraging Metal's low-level GPU access, the new kernels can accelerate real-time image processing, object detection, and segmentation models directly on macOS/iOS hardware. The release also includes CI builds for multiple platforms (Vulkan, CUDA, ROCm) and fixes for F16 kernel support, ensuring cross-platform consistency. For developers building on-device AI apps, this means they can now run hybrid vision-language models more efficiently without needing separate frameworks.

Key Points
  • llama.cpp b9939 adds native Metal CONV_2D_DW (depthwise convolution) support for Apple Silicon, enabling efficient vision model inference.
  • Uses 3D dispatch and channel-tiled kernels to optimize non-contiguous tensor layouts on Metal GPUs.
  • Includes F16 depthwise convolution tests on CPU and cross-platform builds (Vulkan, CUDA, ROCm, OpenVINO).

Why It Matters

Empowers on-device vision AI on Apple hardware, critical for edge inference in AR, photography, and multimodal apps.

📬 Get the top 10 AI stories daily