DeepSeek V4 Flash runs 100 tok/s on four RTX 3060s
DeepSeek-V4-Flash-0731 achieves 99.4 tok/s prompt processing on consumer GPUs with 360k context!
DeepSeek's DeepSeek-V4-Flash-0731, a massive 144 GiB mixture-of-experts (MoE) model, has been successfully benchmarked on consumer hardware, achieving 99.4 tokens/second (tok/s) during prompt processing and 10.1 tok/s during text generation. The feat was accomplished using four NVIDIA RTX 3060 12GB GPUs (totaling 48GB VRAM), an Intel Core i9-10920X CPU, and 128GB DDR4-3200 RAM, with the model loaded via llama.cpp's GGUF format.
The setup leveraged unconventional tensor splitting (-ts 100,1,1,1) and non-contiguous MoE layer distribution (-ncmoe 34) to efficiently allocate the 144 GiB model across GPUs. By pushing non-expert tensors to GPU0 and distributing expert layers across GPUs 1-3, the configuration maximized available VRAM for the largest layers. Performance scaled with microbatch size (-ub 2048), while context length adjustments (368,640 tokens) balanced speed and memory margins.
- DeepSeek-V4-Flash-0731 achieves 99.4 tok/s prompt processing and 10.1 tok/s generation on four RTX 3060 12GB GPUs with a 360k context window
- The 144 GiB MoE model uses llama.cpp with tensor splitting (-ts) and non-contiguous MoE layer distribution (-ncmoe) for optimal performance
- Microbatch size (-ub 2048) and context length (368,640 tokens) are critical for balancing speed and memory constraints
Why It Matters
Proves high-performance AI inference is possible on consumer GPUs, democratizing access to large MoE models for developers.