vllm.cpp: vLLM's server rebuilt in C++20 — 66 MiB, no Python, token-identical
A 66 MiB binary that matches vLLM token-for-token and edges it in speed
A developer has released vllm.cpp, an independent C++20 rewrite of the vLLM serving stack designed to drop the Python dependency entirely. The core goal is embedding inference directly into other software without carrying a 9.1 GiB virtualenv or an interpreter in the process. Every architecture is verified token-for-token against a pinned vLLM oracle on the same workload, with vLLM's own test suite ported alongside the code. The project builds to a 66 MiB binary with no Python and no PyTorch at runtime, supporting continuous batching, block-paged KV cache, prefix caching, and an OpenAI-compatible server.
On a DGX Spark with Qwen3.6-27B NVFP4, vllm.cpp ties vLLM across concurrency levels 2–32 (within noise) and takes a slight 1.045x lead at concurrency 1. Peak GPU memory is 40,996 MiB versus vLLM's 70,531 MiB, though vLLM pre-reserves memory while vllm.cpp allocates on demand. It also hits 1.18x llama.cpp's prefill on CPU aarch64, 97.6% of MLX-LM warm performance on M4, and 18.69 tok/s for DeepSeek-V4-Flash 2-bit GGUF. Speculative decoding gives big wins: MTP boosts c1 from 9.97 to 15.10 tok/s and DFlash from 10.16 to 29.32 tok/s. Supported formats include safetensors, GGUF, NVFP4, k-quants, i-quants, fp8, and bf16, with CUDA, CPU (AVX-512, Arm i8mm), Metal, and partial Vulkan backends.
- Delivers vLLM's full serving stack as a 66 MiB C++20 binary with no Python or PyTorch at runtime
- Output checked token-for-token against vLLM for 25+ architectures; matches or slightly beats vLLM across concurrency 1–32
- Supports speculative decoding (MTP, DFlash), GGUF/safetensors, NVFP4, k-quants, fp8, bf16, and CUDA, CPU, Metal, Vulkan backends
Why It Matters
Proof that production-grade LLM serving can be embedded without Python, enabling smaller, faster, and more secure deployments.