b8485

The open-source project llama.cpp, maintained by the ggml organization, has released a significant server optimization in its latest commit (b8485). This update fundamentally changes how the built-in HTTP server handles concurrent connections by switching from a fixed thread pool to a dynamic threading model using httplib. The new system allocates threads based on the formula `n_threads_http + 1024`, which allows the server to scale more efficiently under load, particularly for applications with fluctuating numbers of simultaneous API requests, such as those powering chatbots or batch processing jobs.

This technical improvement is part of the continuous performance tuning for the widely used inference engine, which boasts 99.1k stars on GitHub. The commit is bundled within pre-built binaries for a vast array of platforms and hardware backends, ensuring the benefit reaches a broad developer base. Available builds now include versions for macOS (both Apple Silicon and Intel), Linux (with CPU, Vulkan, ROCm 7.2, and OpenVINO support), Windows (with CPU, CUDA 12/13, Vulkan, SYCL, and HIP), and openEuler (with Huawei Ascend NPU support via ACL Graph). This cross-platform availability means developers deploying local LLMs—from researchers to application builders—can immediately leverage more responsive and scalable server behavior without changing their application code.