Qwen 3.6 vs Gemma 4: Quantization trade-off for local coding AI
A Reddit user pits Q4 Qwen 35B against Q8 Gemma 12B on 32GB Mac memory.
Deep Dive
A user with 32GB unified memory runs a Qwen model at ~15 tokens/second, is curious about the 12B Gemma 4 for code tasks, and notes it can run at Q8 or even BF16—wondering how much model quantization actually matters.
Key Points
- Qwen 3.6 35B-A3B at Q4 runs at ~15 tokens/sec on 32GB unified memory, but quantization may hurt reasoning.
- Gemma 4 12B can fit at Q8 or BF16, offering full precision but fewer parameters for code tasks.
- Trade-off highlights common decision point for local AI users: model size vs quantization depth vs speed.
Why It Matters
For developers running models locally, choosing the right quant matters more than raw parameter count.