Hobbyist adds vision to DeepSeek V4 Flash with 40M connector for $2K

A solo developer demonstrated that a massive text-only MoE can gain basic multimodal capability without retraining the language model. Using DeepSeek V4 Flash (284B total / 13B active parameters) as the frozen backbone and MoonViT-3d, a 417M-parameter image encoder from Kimi K2.6, they trained only a 40,119,040-parameter connector on 100,000 image-text examples sampled from HuggingFaceM4/the_cauldron. The pipeline converts images to 1152-dim features, merges 2×2 patch groups, and maps them through a small MLP into 4096-dim embeddings for the frozen LLM. With a global batch size of 128, AdamW at 1e-3, and BF16 precision, training took roughly 20.25 hours on 5× H200 GPUs at ~4,938 examples/hour. The final loss dropped from 1.0225 to 0.8301, and the full project—including caching, conversion, and inference validation—cost about $2,000.

To avoid repeatedly running the 417M encoder, the developer cached MoonViT outputs for all 39,619 unique images (the dataset had multiple questions per image). The completed model, released as DeepSeek V4 Flash Vision NVFP4, loads across four B200s in a custom SGLang stack and can answer real image prompts. In one test, it recognized people, boxes, and a storefront sign from a street image. The author notes this 100K-example pilot is not production-ready, but it proves that cheap, targeted connector training can unlock basic vision in frozen frontier-scale MoEs. Weights are available on Hugging Face in both NVFP4 and BF16 formats.