Tensor split boosts Qwen3.6-27B Q8 to 100+ t/s on 5090 + 3090 Ti

A Reddit user documented a high-performance local LLM inference setup using llama.cpp to run Qwen3.6-27B (the latest Qwen variant) at Q8_0 quantization across two heterogeneous GPUs: an RTX 5090 and an RTX 3090 Ti. The key breakthrough came from switching --split-mode from the default layer split to tensor split. In layer split mode, each GPU handles entire layers sequentially, causing idle time on the slower card. Tensor split mode, by contrast, partitions each tensor across both GPUs, keeping both cards busy simultaneously on the same computation. With a 70/30 tensor split ratio favoring the more powerful RTX 5090, the user achieved around 100 tokens per second, up from the 70+ t/s seen with layer splitting. Peak performance reached 130 t/s on certain prompts.

The system consumes over 750 watts from the GPUs alone during decoding, turning the setup into a “space heater.” The full llama.cpp server command includes flags for offloading 99 layers to GPU, disabling memory mapping, using mlock, setting a massive 196,608 token context, and enabling speculative decoding with a draft model (--spec-type draft-mtp). The user also specified NUMA isolation, 24 CPU threads, and a reasoning budget of 12,288 tokens. This configuration demonstrates how careful multi-GPU tuning—especially tensor split mode—can dramatically improve throughput for large language models on consumer hardware.