Gemma 4 QAT beats Qwen 3.6 in real-world performance despite lower benchmarks
Benchmark scores don't tell the whole story for local LLMs.
A fascinating comparison has emerged in the local LLM community: despite Qwen 3.6 35a3B posting vastly superior benchmark scores, users report that Gemma 4 26a4B (with QAT) feels 'substantially less intelligent' in practice. The Reddit user /u/Jon_vs_Moloch notes that on actual tasks—prompt adherence, output coherence, and general 'sanity'—Gemma appears head-and-shoulders ahead. This gap is echoed by Arena.ai rankings, where Gemma 4 26a4 ranks only about 7 ELO points below the larger Qwen 3.6 Plus proprietary model. Both models were run at Q4 quantization, so quantization likely affects both, but the user suspects Qwen's QAT might be the key factor making Gemma feel smarter.
This discrepancy between synthetic benchmarks and real-world utility highlights a critical challenge for the AI community: raw scores don't always translate to usable intelligence. The poster asks if others have seen similar patterns and whether there's a trick to make Qwen act smarter. The implication is that for local deployment where resources are constrained, model architecture and training techniques like QAT may matter more than benchmark bragging rights. As local LLMs become more practical, developers and hobbyists may need to rely on human evaluation over leaderboard positions.
- Gemma 4 26a4B (QAT) felt 'substantially more intelligent' than Qwen 3.6 35a3B in real usage despite Qwen having higher benchmark scores.
- Arena.ai shows Gemma only ~7 ELO behind Qwen's larger proprietary model, suggesting practical performance over benchmarks.
- User ran both at Q4 quantization; suspects Qwen's quantization-aware training (QAT) may explain Gemma's superior coherence.
Why It Matters
Benchmark scores may mislead—real-world usability for local LLMs depends on more than raw metrics.