Llama.cpp b9716 Unlocks Batching for InternVL — Here's the Performance Boost You've Been Waiting For

The llama.cpp project, a popular open-source C++ library for running large language models locally, has released version b9716. The headline feature is the addition of batching support for InternVL, a family of open-source multimodal models that combine vision and language understanding. Previously, InternVL inference was limited to single-input processing; with batching, users can now feed multiple images and text prompts concurrently, dramatically improving performance for tasks like batch image captioning, document analysis, or multimodal retrieval. The release also includes numerous platform-specific builds and bug fixes (e.g., mtmd improvements).

This update reinforces llama.cpp's position as the leading solution for on-device AI. With over 117,000 stars and nearly 20,000 forks, the project supports a vast array of hardware, from Apple Silicon and Intel Macs to Windows with CUDA 12/13, Vulkan, and even ROCm for AMD GPUs. By adding batching for multimodal models, llama.cpp enables developers and researchers to run advanced vision-language models efficiently on consumer hardware, reducing reliance on cloud APIs. This is a significant step toward democratizing multimodal AI inference for privacy-sensitive or offline applications.