Qwen 3.6 quantization cripples agentic performance but spares knowledge
Lower precision models drop 40%+ on agentic benchmarks while GPQA scores remain flat.
Researchers managing a university HPC cluster tested Qwen 3.6 quantizations (including FP8 and lower-precision variants) on two benchmarks: GPQA Diamond for knowledge and Terminal-Bench 2 for agentic capabilities. The results confirm that quantization barely affects knowledge retention — GPQA scores remain nearly identical across all precisions. However, agentic performance (task completion, tool use) shows a clear regression as precision drops, with the worst quantizations scoring significantly worse.
The team also discovered that timeout settings dramatically influence results. Qwen’s official FP8 scores used a flat 3-hour timeout, while the university’s setup with Harbor’s default (10–60 minutes per task) produced much lower scores — sometimes 30–50% lower. This suggests agentic benchmarks are highly sensitive to time limits. Additionally, individual run variance is large: a lucky low-precision run can sometimes beat an unlucky high-precision run. The team is now benchmarking GLM-5.2 quantizations.
- GPQA Diamond scores remain nearly constant across all Qwen 3.6 quantizations — knowledge retention is robust.
- Terminal-Bench 2 agentic scores drop sharply at lower precisions, with FP8 showing up to 40% regression.
- Timeout settings cause major score differences: Qwen’s 3-hour timeout vs Harbor’s 10–60 min default changes results significantly.
Why It Matters
Professionals deploying quantized models for agentic tasks must re-evaluate precision trade-offs and benchmark conditions.