Qwen3.8-27B hits 218 tok/s on 2x RTX 3090 with vLLM
A custom vLLM setup pushes Qwen3.8-27B to 218 tok/s on dual 3090s
A Reddit user (xjx546) hacked together an optimized serving stack for Qwen3.8-27B on dual RTX 3090s, achieving 218 tok/s single-request decode — over 2x faster than the baseline 120 tok/s. The setup pairs bare-metal vLLM v0.26.1rc1 with AutoRound INT4 quantization (group 128) and a DFlash2 draft model for speculative decoding. With 7 draft tokens, they measured 47.8% acceptance and an average acceptance length of 3.35, yielding 204.8 tok/s wall-clock decode. Prefill hits 1342 tok/s at 10k context and 628 tok/s at 90k, while peak VRAM stays at 22.3 GB per card — leaving headroom on the 24 GB 3090.
The custom vLLM changes needed to boot cleanly are open-sourced on GitHub via a pull request from oceanplexian. The context ceiling is 131k tokens, despite the DFlash2 drafter consuming ~13.5 GB of VRAM. The system runs on PCIe Gen4 x16/x16 without NVLink, with patched P2P and a 220/250W power cap. This is a significant achievement for consumer-grade hardware, showing that with careful quantization and speculative decoding, frontier-level open models can be served efficiently at home — though xjx546 admits there's likely more performance on the table.
- 218 tok/s decode speed single request, up from 120.1 tok/s baseline
- Speculative decoding with DFlash2: 7 draft tokens, 47.8% acceptance rate
- Peak VRAM 22.3 GB/card, context ceiling 131k tokens on 2x RTX 3090
Why It Matters
Proves high-throughput local LLM serving is feasible on commodity GPUs with the right quantization and decoding tricks.