Google's Gemma4 QAT delivers 2.4x speed boost and quality improvements

A detailed user report on Google's Gemma4 QAT (Quantization-Aware Training) model highlights substantial practical benefits for local AI inference. The user, primarily using Gemma4 for non-programming tasks like translation, OCR, and long-document parsing, switched from separate quantized versions (Q4_K_L for 128K context and Q6_K_L for 32K) to a single QAT model. They observe subtle quality improvements, especially in roleplay scenarios—more varied word choice, better contextual awareness, and improved correlation understanding. Performance-wise, enabling Multi-Token Prediction (MTP) with 5 draft tokens yields impressive throughput: 50 tokens/s for 32K wikipedia summarization (up from 21 t/s) and 36 t/s during interactive roleplay (up from 20 t/s). The user notes that cache quantization does cause noticeable degradation at 128K context, suggesting the model excels at shorter contexts. They also recommend experimenting with draft token counts between 3-7 to find the optimal setting for specific tasks.

The report underscores the growing maturity of quantization techniques for large language models. Gemma4 QAT enables users to run a single, high-quality 31B parameter model locally with consumer hardware, achieving near-realtime inference speeds. The combination of QAT and MTP effectively narrows the gap between cloud-based and local AI workloads, making advanced language tasks like research summarization and interactive storytelling feasible on mid-range GPUs. While programming performance isn't covered, the user's positive experience suggests Gemma4 QAT is a strong all-around choice for professionals needing efficient local inference with nuanced output.