This Tiny Addition in Llama.cpp v0.0.0-b9714 Instantly Fixes Nginx Streaming Issues
New header prevents Nginx from breaking AI model streaming responses.
The open-source AI inference engine llama.cpp has released version b9714, a patch that addresses a critical streaming issue when used behind Nginx reverse proxies. The update adds the “X-Accel-Buffering”: “no” header to all streaming endpoints, explicitly instructing Nginx not to buffer responses. Without this header, Nginx would accumulate the entire response before forwarding it, breaking real-time streaming behavior that is essential for chat and code-generation applications. The fix was specifically motivated by issues encountered with the Pi coding harness, a tool that relies on continuous streaming of model outputs.
The release, signed with GitHub’s verified GPG key, provides pre-built binaries across an extensive range of platforms: macOS (Apple Silicon and Intel), iOS XCFramework, Linux (x64, arm64, s390x with CPU, Vulkan, ROCm 7.2, OpenVINO, and Intel SYCL FP32/FP16), Windows (x64/arm64 CPU, CUDA 12.4 & 13.3 DLLs, Vulkan, OpenVINO, SYCL, and HIP), Android arm64, and openEuler variants. This update is particularly important for developers deploying llama.cpp behind Nginx, as it ensures low-latency, real-time token generation without buffering delays. The fix is simple but impactful—a single header change that restores proper streaming behavior for all downstream applications.
- Added 'X-Accel-Buffering: no' header to all streaming endpoints in llama.cpp b9714
- Prevents Nginx from buffering responses, fixing streaming for Pi coding harness and other tools
- Pre-built binaries available for macOS, Linux, Windows, Android, and openEuler with multiple acceleration backends
Why It Matters
Ensures real-time AI streaming works behind Nginx, critical for production LLM serving and coding assistants.