New speculative decoding method doubles inference speed on Qwen3.6-27B
Exploits Q8 compute inefficiency to run same model as draft and target.
Reddit user bigattichouse is developing a speculative decoding trick that uses the same model as both draft and target, eliminating the need for a separate side model. The idea exploits how every Q8 (INT8 or F8) calculation uses f32 compute but only a quarter of the available numbers, so each loaded value can run four operations. The approach only works with smaller quants (Q8 or less). The user is still working on a HIP kernel and combining it with MTP, hoping to get near 80 tk/s. They shared benchmarks from a single AMD MI50 and referenced detailed tables and SVGs in the readme.
- Achieved 2x tokens/second (19.4 → 38.1 tk/s) on single AMD MI50 with Qwen3.6-27B using Q8 quants.
- Technique uses the same model for both draft and verification by exploiting underutilized F32 compute in low-bit operations (4 ops per value).
- Plans to combine with multi-token prediction (MTP) and a custom HIP kernel to target 80 tk/s.
Why It Matters
Potential to dramatically speed up LLM inference on consumer GPUs without needing extra models or memory.