Qwen3.6-27B beats tuned 75B agent with 3x fewer tool calls
Smaller untuned model outperforms massive one on agentic benchmarks — fewer turns, less time.
In a recent real-world test, the Qwen3.6-27B (quantized to INT8, running with vLLM and 131K context) outperformed the much larger Nemotron Puzzle-75B-A9B (NVFP4, across 3 cards) on agentic tasks. The 27B model, without any custom tuning, passed every task using only 6–9 tool calls and completing in 134–190 seconds. In contrast, the 75B model required a hand-tuned system prompt to reliably pass, and even then needed 13–23 tool calls and 221–384 seconds. The 27B achieved 37.7 t/s fresh and ~26 t/s deep context, while the 75B hit ~65 t/s decode. But agentic efficiency depends on minimizing turns, not maximizing token throughput.
An additional discovery during testing: byte-identical agent runs failed 6/6 times due to what appears to be a prefix caching bug in the vLLM server (using fp8 KV cache). The same payload passed 2/2 after cache churn. The suspected culprit is prefix caching serving corrupted cached prefixes. Adding a per-run nonce to the system prompt made the issue unreproducible. This highlights a critical flaw in benchmarking with prefix caching enabled — identical retries are not independent samples, potentially skewing results.
- Qwen3.6-27B (untuned) completed all agentic tasks in 6–9 tool calls vs 75B requiring 13–23 calls even after hand-tuning.
- 27B model ran at ~26 t/s deep context vs 75B's 65 t/s, but total wall time was 30–50% lower due to fewer wasted turns.
- A prefix caching bug in vLLM (fp8 KV) caused identical benchmark runs to fail 6/6 until cache churn resolved it — adding a per-run nonce is recommended.
Why It Matters
Smaller, untuned models can outperform larger ones on agentic tasks when efficiency of tool use matters more than raw token speed.