vLLM v0.23.0 boosts DeepSeek-V4 performance and expands model support
408 commits, 200 contributors, and major optimizations for sparse MLA and MoE architectures.
The vLLM team has shipped v0.23.0, a major release featuring 408 commits from 200 contributors (63 new). DeepSeek-V4, introduced in v0.22.0, receives extensive hardening and optimization: its sparse MLA metadata is now decoupled from DeepSeek-V3.2 (#44699), gaining a TRTLLM-gen attention kernel (#43827), EPLB support for the Mega-MoE (#43339), selective prefix-cache retention for sliding-window KV cache (#43447), and an index-share feature for DSA MTP (#44420). The model was also detached from torch.compile, with refactored attention and RoPE paths (#44569, #44262, #43926) and an XPU attention decode path (#42953). Model Runner V2 (MRv2) expands to more dense models: default for Llama and Mistral (#43458) alongside Qwen3, with a FlashInfer sampler (#42472), breakable CUDA graphs (#44050), pipeline-parallel bubble elimination (#42187), and kernel block-size support for hybrid models (#38831). The experimental Rust frontend now includes streaming generate (#43779), dynamic LoRA endpoints (#43778), /version and /server_info endpoints, server-router extension hooks, request-ID headers, and new tool parsers for InternLM2, hy_v3, Phi-4-mini, and Gemma4.
Other highlights include encoder-free Gemma 4 Unified support (#44429) with numerous accuracy and startup fixes. vLLM targets Transformers v5, with vendored MiniCPM-V/O processors (#44282) and compatibility fixes for Sarvam and Voxtral. A multi-tier KV cache offloading framework adds an object-store secondary tier (#41968), HMA enabled by default for capable connectors (#41847), tiering support for HMA models (#44287), and per-request offloading policies. Reasoning and tool-call parsing are unified behind a single Parser.parse() interface (#44267). New models include Step-3.7-Flash (#43859), Cosmos3 Reasoner (#43356), Granite Speech Plus (#43519), Cohere Mini Code (#44707), JetBrains Mellum v2 (#43992), and Gemma 4 MTP (#43241). Numerous fixes target Qwen3-VL, GLM, MiniCPM, and other architectures.
- DeepSeek-V4 gets sparse MLA decoupling, TRTLLM-gen attention kernel, and EPLB support for Mega-MoE, plus detached from torch.compile.
- Model Runner V2 becomes default for Llama and Mistral dense models, adding FlashInfer sampler and breakable CUDA graphs.
- Multi-tier KV cache offloading adds object-store secondary tier and per-request offloading policy, improving memory efficiency.
Why It Matters
vLLM v0.23.0 delivers critical inference optimizations for sparse MoE models, expanding production-ready support for DeepSeek, Gemma, and more.