Gemma 4 12B QAT hits 120 tok/s on 12GB GPU with MTP
Google's Gemma 4 runs 2x faster on consumer hardware using speculative decoding.
A developer benchmarked Google's newly released Gemma 4 12B QAT (Quantization-Aware Training) model on modest consumer hardware — an RTX 4070 Super with 12GB VRAM — achieving an impressive 120 tokens per second. The trick was using a custom llama.cpp build patched with Gemma 4's Multi-Token Prediction (MTP) support, alongside Unsloth's GGUF quantized version of the model and a specially converted draft/assistant model (Q8_0 GGUF) from Google's original. The aggregate benchmark across nine tasks (including code, QA, summarization) showed draft acceptance rates ranging from 46% to 82%, with an overall accept rate of 65.78%.
Without MTP, the model ran at a steady ~60 tok/s. With MTP, it doubled to 120 tok/s aggregate, and individual tasks like stepwise math hit 135.7 tok/s. The setup requires a 12GB NVIDIA GPU and enough free VRAM to load both the main model and the draft model. The user's CachyOS Linux setup with the dGPU as secondary freed up essentially all VRAM; on Windows or with a primary dGPU, expect 500MB+ overhead, potentially requiring reduced context size or lower quantization.
The significance: this is the first time a 12B parameter model has demonstrated near-real-time token generation on a broadly available GPU. For reference, most local LLM users consider 30-50 tok/s as usable; 120 tok/s approaches the speed of cloud-based APIs. The method — speculative decoding via MTP — is already present in llama.cpp but requires model-specific support. Google's QAT training and the availability of a compatible draft model made this possible. Replication steps are straightforward: download the two GGUF files, run llama-server with the --spec-type draft-mtp flag, and benefit from a ~2x speedup.
While the benchmark is impressive, it's a single user's test on a specific OS and GPU. Real-world performance may vary. Still, it signals that local AI inference with large models is rapidly approaching practical speeds for interactive use, even on hardware consumers already own. This could accelerate adoption of local AI assistants, code completions, and content generation without cloud dependencies.
- Achieved 120 tok/s on a 12GB RTX 4070 Super using Google's Gemma 4 12B QAT model with Multi-Token Prediction (MTP)
- 2x speedup over baseline (60 tok/s) with a 65.78% aggregate draft acceptance rate across 9 tasks
- Requires llama.cpp with MTP patch, two GGUF files (main + draft), and ~12GB free VRAM; Linux recommended to avoid OS VRAM overhead
Why It Matters
Brings fast, local 12B-parameter inference to consumer GPUs, reducing reliance on cloud APIs.