llama.cpp thread tuning: +80% speed boost with 16 threads

A recent experiment with llama.cpp on an Intel Core 250K Plus CPU (6 performance cores + 12 efficiency cores) revealed that adjusting the --threads argument can dramatically improve inference performance. The user tested Gemma 4 26B A4B QAT (quantized from Unsloth) with MTP draft models, varying thread counts from 6 to 18 while keeping all other settings identical. Results showed a clear sweet spot: 16 threads delivered 88.8 tokens/second average, an 80% uplift over the 6-thread baseline (49.1 tok/s). Surprisingly, using all 18 cores (including E-cores) slightly reduced performance (66.5 tok/s), suggesting a bottleneck elsewhere—possibly memory bandwidth or cache contention. The user noted that previous conventional wisdom favored restricting threads to P-cores only, but Arrow Lake's thread director may handle hybrid scheduling better than older Raptor Lake CPUs.

This finding challenges the common practice of limiting inference threads to P-cores for hybrid CPUs. The developer used a rigorous test: same prompt, same seed, one warmup run followed by five measurement runs. The optimal command also included adjustments to MTP draft count (--spec-draft-n-max 2 instead of 3) and full GPU offloading on an RTX 4070 SUPER. The takeaway is clear: users should benchmark thread counts on their own hardware rather than relying on outdated advice. Even small configuration tweaks can yield substantial speedups—potentially halving response times for local LLM applications. The post also highlights the importance of testing every setting with your specific model and hardware combination.