Kimi K3 runs at 4 t/s on home lab with 2x5090 and 768GB DDR5
A Reddit user achieves 4 t/s decode on Kimi K3 using a custom llama.cpp fork and Q2_K quant.
In a surprising computational feat, Reddit user u/iVoider shared preliminary results from running the Kimi K3 model on a home-built workstation. The setup includes 768GB of DDR5 RAM and two NVIDIA GeForce RTX 5090 GPUs, pushing the boundaries of local large language model inference. Using a specialized fork of llama.cpp by pwilkin (kimi-k3-text branch) and a Q2_K quantized GGUF file from GrEarl on Hugging Face, the user achieved a decode speed of roughly 4 tokens per second (t/s), with prefill reaching 50-70 t/s for large prompts. Notably, the decode throughput appeared to increase over the course of the session, hinting at either memory swap optimization or GPU/kernel warm-up effects. The llama-bench tool crashed, preventing standardized benchmark comparisons.
This development is significant because Kimi K3 is understood to be a very large model—potentially 100B+ parameters—originally designed for cloud deployment. Running it on consumer-grade hardware with quantization demonstrates the rapid progress in model compression and inference software. While 4 t/s is far slower than cloud APIs, it opens possibilities for privacy-preserving local inference of state-of-the-art models. The reliance on 768GB system memory suggests the model may be too large to fit in VRAM even with two 5090s (each has 32GB VRAM), so it heavily leverages system RAM via CPU offloading. Future optimizations or faster NVLink could improve speeds.
- Kimi K3 runs at ~4 t/s decode and 50-70 t/s prefill on a home rig with 768GB DDR5 and 2x5090 GPUs.
- The user used a custom llama.cpp fork and Q2_K GGUF quantization to fit the model into memory.
- Decode speed improved over time, possibly due to memory swap or GPU warm-up effects; benchmark tool crashed.
Why It Matters
Proves massive models like Kimi K3 can run locally on enthusiast hardware, advancing private AI inference.