Speculative decoding question, 665% speed increase

A viral Reddit post has highlighted the transformative potential of speculative decoding, a technique for accelerating large language model inference, when applied to real-world coding tasks. The user, experimenting within the popular llama.cpp framework, shared a specific configuration that yielded extraordinary results: a 665% speed increase when using the Devstrall Small model for generating 'minor changes in code.' The key settings included `--spec-type ngram-map-k`, `--spec-ngram-size-n 24`, and draft token ranges (`--draft-min 12 --draft-max 48`), which guide a smaller, faster draft model to predict likely token sequences for verification by the primary, more capable model.

The performance gains were not uniform across all models, revealing important nuances in how different architectures respond to the technique. While the Devstrall Small model saw the staggering 665% boost, Google's Gemma 2 9B doubled its token generation speed (a 100% increase), and Qwen 3.6 initially showed a more modest 40% gain. A follow-up edit showed that tweaking settings further—switching to `--spec-type ngram-mod` and adjusting the `--repeat-penalty`—could push Qwen 3.6's performance to a 140% increase over its baseline. This variance underscores that speculative decoding is not a one-size-fits-all optimization but requires model-specific tuning to unlock its full potential, especially for domain-specific tasks like code iteration where token sequences are more predictable.