DiffusionGemma 26B hits 100 tps on four AMD 7900 XTX GPUs

In a breakthrough for AMD-based AI inference, a Reddit user (djdeniro) demonstrated the large language model DiffusionGemma 26B running on four AMD Radeon RX 7900 XTX GPUs using the vLLM serve framework. The system achieves a generation speed of 100 tokens per second (tps), though effective throughput including prompt processing lands at 45–60 t/s. Each GPU uses nearly all of its 24 GB VRAM (23.6/24.0 GB) and draws between 154–183W of power, with temperatures hovering around 80–88°C. The model supports a context window of 131,072 tokens with a GPU KV cache size of 152,671 tokens, and a maximum concurrency of 1.16 requests per token budget.

The setup relies on a heavily customized Docker image built on the vllm-dgemma branch, with environment variables like VLLM_USE_V2_MODEL_RUNNER=1 and attention backend set to TRITON_ATTN. The user reported that preparing this Docker image consumed approximately 2–3 million tokens from DeepSeek-v4-pro, highlighting the high experimentation cost. Despite that, the result is one of the first public demonstrations of a 26B-parameter diffusion-style Gemma model running efficiently on consumer AMD hardware. It signals growing viability for AMD GPUs in large model inference, potentially lowering the barrier for teams seeking alternatives to Nvidia hardware for self-hosted AI workloads.