Google DeepMind's DiffusionGemma runs local AI 4x faster with parallel text generation

Google DeepMind has unveiled DiffusionGemma, a fundamentally different kind of text model that generates outputs in parallel rather than token-by-token. Unlike standard autoregressive models that predict one token left-to-right, DiffusionGemma starts with a field of placeholder tokens and iteratively denoises them over multiple passes, finalizing up to 256 tokens in one block. This approach shifts the bottleneck from memory bandwidth to compute, delivering up to 700 tokens per second on an RTX 5090 and over 1,000 tokens per second on an Nvidia H100—roughly four times the throughput of similarly sized Gemma autoregressive models. The model is a Mixture of Experts design with 26 billion total parameters but only 3.8 billion activated per inference, allowing it to run within 18GB of GPU memory, suitable for high-end consumer GPUs.

DiffusionGemma excels at non-linear tasks like in-line text editing, molecular sequencing, and solving Sudoku puzzles, where each token depends on future tokens. However, it has downsides: a higher error rate compared to autoregressive models, as a single bad token can corrupt an entire block, and inefficiency for generating very short outputs. Google emphasizes it's experimental but releases it under Apache 2.0 on Hugging Face, optimized in collaboration with Nvidia for both RTX GPUs (quantized) and enterprise systems like DGX Spark. While cloud AI already uses batching to keep GPUs busy, local AI suffers from idle compute cycles. DiffusionGemma and Google's Multi-Token Prediction drafters both aim to fill those cycles, but diffusion offers even greater speedups for local deployment.