Llama.cpp fork delivers 40% faster inference on dual GPUs with tensor splitting fix

A Reddit user, Legitimate-Dog5690, has released a fork of llama.cpp that fixes a longstanding issue with tensor parallelism. The original '--split-mode tensor' option offered great performance but only supported non-quantized KV caches, forcing many users to stick with larger KV caches instead of utilizing dual GPUs. The fork, branched from the mainline as of today with minimal changes, enables quantized KV caches (q8_0) with tensor splitting, unlocking a significant speedup.

Benchmarks on the Qwen3.5 27B Q4_K_M model with a 3060 (12GB) + 4070 Super (12GB) show a 40% improvement in tokens per second: 30.05 t/s with tensor split versus 21.22 t/s without. For prompt processing, the fork achieves 544.82 t/s, close to the 582.60 t/s of single-GPU mode. In real-world usage, the user reports going from ~25 t/s to ~40 t/s in story-generation contexts, aided by MTP speculative decoding (e.g., `--spec-type draft-mtp`). The fork also supports the latest multi-token prediction changes.

Key limitations: MoE (Mixture of Experts) models have an unrelated issue with tensor splitting, so dense models like Qwen3.5 27B or 9B are recommended. The user invites feedback from those running dual 5060 Ti or Vulkan setups. If popular, they plan to fix MoE support and integrate Turboquants. Overall, this is a free, immediate performance boost for anyone with dual GPUs who runs local LLMs via llama.cpp.