b8466

The llama.cpp project, a leading C++ framework for running large language models locally, has patched a significant bug in its latest release. Commit b8466 specifically addresses a tensor out-of-bounds assertion error that occurred when using the embedding functionality with Alibaba's Qwen3-VL-Embedding models. The issue stemmed from the code using an incorrect tensor dimension (n_embd_inp, which is 16384 for Qwen3VL) to read from a pooled embedding tensor that only contained a smaller number of floats (n_embd_out, 4096). This fix is crucial for the 'embedding mode,' which is essential for retrieval-augmented generation (RAG) and other tasks that rely on converting text or images into numerical vectors.

The fix, labeled 'context: use n_embd_out for pooled embedding extraction,' is now part of the stable builds available across all major platforms. This means developers and researchers can reliably run the 72-billion-parameter Qwen3-VL model and its embedding variants on their own hardware. The supported ecosystems are extensive, including macOS on Apple Silicon and Intel, iOS, various Linux distributions (with CPU, Vulkan, and ROCm backends), and Windows (with CPU, CUDA 12/13, Vulkan, and SYCL support). This patch underscores the rapid, community-driven development of the local AI inference ecosystem, making state-of-the-art multimodal models accessible without cloud dependencies.