DFlash2 turbocharges Qwen3.8-27B on RTX 3090
RTX 3090 users now get 138 tokens/sec with Qwen3.8-27B using DFlash2 optimizations
A community developer has released **DFlash2**, an inference engine that dramatically accelerates **Alibaba's Qwen3.8-27B** model on consumer GPUs like the RTX 3090. The update builds on prior optimizations (fp8 KV cache, int8 quantization, and MTP-4 drafting) with new techniques like **DFlash2 drafting** and **lookup-augmented drafting**, pushing single-user throughput to **138 tokens/sec** (from 82 tps) and multi-turn latency down to **~1 second** from 23 seconds.
Key innovations include a **W4A16 quantized drafter** (reducing memory to 1.19GB with Hessian-captured weights) and **lookup-augmented drafting**, which scans prompt history to predict token sequences with **+29% tokens/step**. The engine also adds **prefix caching** for hybrid models, slashing latency on long documents from 23s to 0.85s per turn. vLLM integration remains experimental but promises even further gains.
- DFlash2 achieves **138 tps** on Qwen3.8-27B (RTX 3090), up from 82 tps, using quantized drafting and lookup-augmented generation
- New features: W4A16 drafter (1.19GB), lookup-augmented drafting (+29% tokens/step), and prefix caching (23s → 0.85s per turn)
- vLLM integration (unmerged PR) enables batch processing: 64 requests with a 5,820-token prompt drop from 222s to 16.9s median latency
Why It Matters
Unlocks near-real-time LLM inference on mid-range GPUs, making advanced AI accessible to budget-conscious developers and researchers.