Deepseek V4 Flash runs on a single RTX 6000 Pro with custom vLLM-Moet
Custom compression lets a 671B MoE model run on one consumer GPU.
A developer known as kacper-daftcode has released vLLM-Moet, a custom fork of vLLM that compresses Deepseek V4 Flash's routed experts to 2-bit precision while retaining fp4 for other experts. This technique allows the 671B-parameter Mixture-of-Experts model to fit entirely in the 48 GB VRAM of a single NVIDIA RTX 6000 Pro (and reportedly works on a single RTX 5090). During loading, the system needs roughly 150 GB of system RAM to handle safetensor sharding; once in VRAM, it runs entirely on the GPU. The model supports up to 130K context tokens.
Benchmarks run by the user show solid performance: 128 tokens per second on text generation (tg32), and around 110 t/s with a 8K context length. Time-to-first-token (TTFT) varies from 5–17 seconds depending on context size. The release also supports GLM 5.2, though that requires two RTX 6000 Pros. While still early—users should test coding and reasoning quality—this is a remarkable feat of model compression that brings state-of-the-art MoE models to affordable single-GPU setups.
- Deepseek V4 Flash runs on a single RTX 6000 Pro (48 GB VRAM) using vLLM-Moet's 2-bit routed experts + fp4 other experts.
- Loads require ~150 GB system RAM, but once in VRAM, achieves 128 t/s generation and supports 130K context.
- Also compatible with GLM 5.2 on two RTX 6000 Pros, expanding local AI capabilities.
Why It Matters
Brings 671B MoE models to affordable single-GPU setups, democratizing local high-performance AI inference.