DeepSeek V4 Flash runs 1M token context on RTX 5090 with new CUDA patch
From 256GB VRAM to 31GB: llama.cpp patch unlocks 1M context locally.
A developer has drastically reduced the hardware requirements for running DeepSeek V4 Flash locally. Previously, attempting 1M token context on llama.cpp demanded an absurd ~256GB of VRAM due to a missing CUDA path and improper wiring of the DSA lightning indexer. Spencer Zaid (u/da_dragon321) implemented a custom CUDA kernel and integrated it into the model graph, enabling the quantized Q8/Q4/Q2 version by antirez to run on a single RTX 5090 (32GB) with just 31GB peak VRAM at 1M context.
The patch delivers impressive performance: prefill at 159 t/s for 1M tokens (or 263 t/s at 256K context) and decode at a consistent 13.7 t/s. Accuracy was verified with needle-in-haystack tests at 100K, 512K, and 1M lengths – the model correctly retrieved planted facts at 10%, 50%, and 90% depth. The source, build instructions, and full documentation are available on GitHub. While no prebuilt binary is provided, the instructions are straightforward for users comfortable with compiling llama.cpp.
- Custom CUDA kernel reduces 1M token context VRAM from ~256GB to 31GB on RTX 5090.
- Achieves 159 t/s prefill and 13.7 t/s decode at 1M context with validated accuracy.
- Patch supports DeepSeek V4 Flash quantized by antirez; code available on GitHub for custom builds.
Why It Matters
Brings frontier-level 1M token context to consumer GPUs, enabling local long-document analysis and research.