DeepSeek V4 Flash gets early llama.cpp support, rivals frontier models locally
First local model in its size range to feel comparable to frontier models.
An early pull request (PR #24162) on llama.cpp is bringing support for the DeepSeek V4 Flash series to local inference. The PR is still very early—GPU and Flash Attention support are incomplete, and performance hovers around 5-6 tokens per second. However, early experimenter Lowkey_LokiSN manually quantized the model to a custom 3-bit format and found that the model's intelligence for its size is unprecedented. They claim it's the first time a local model in the 80-140GB range actually feels comparable to frontier cloud models.
The key to this success is DeepSeek V4 Flash's native FP4-FP8 hybrid architecture, which fares much better against quantization than traditional FP16 models. Unlike MiniMax M2.7, which degrades even with UD-Q4_K_XL quantization, DeepSeek V4 Flash retains quality even at 3 bits. Additionally, its KV cache consumption is dramatically lower without needing flash attention, outperforming even the popular Qwen 3.5/3.6 series in memory efficiency. Contributors fairydreaming (DSA implementation), am17an, and pwilkin are driving this effort, and the community expects this model to dominate local inference for months once merged.
- Experimental llama.cpp PR #24162 enables DeepSeek V4 Flash at 5-6 tps with custom 3-bit quantization
- Native FP4-FP8 hybrid architecture resists quantization degradation far better than FP16 models like MiniMax M2.7
- KV cache is extremely efficient, consuming far less memory even without flash attention, outpacing Qwen 3.5/3.6
Why It Matters
Brings frontier-level intelligence to local hardware with minimal memory cost and quantization loss.