The Low-End Theory! Battle of < $250 Inference

A detailed, viral benchmark from Reddit user m94301 provides a crucial cost-to-performance analysis for hobbyists and developers building budget local AI inference rigs. The test compared five used GPUs—the Tesla P4 (8GB), CMP170HX (10GB), RTX 3060 (12GB), CMP100-210 (16GB), and Tesla P40 (24GB)—all sourced for under $250 each on eBay. Using llama.cpp with the '-ngl 99' flag to maximize GPU offloading, the benchmark ran six quantized models, from the 2.3GB Qwen3-VL-4B to the 14.6GB Codestral-22B, measuring tokens/second.

The results reveal a clear hierarchy. For models under 12GB, the CMP100-210 (a mining card) and the consumer RTX 3060 consistently led in raw speed, with the CMP100-210 hitting 91.44 tokens/sec on Mistral-7B. However, the Tesla P40, despite slower speeds on smaller models, was the only card that could load the massive 22B parameter model, achieving 12.09 tokens/sec. This highlights the critical trade-off: speed versus memory capacity. The Tesla P4, while the cheapest per GB at $10.13, often couldn't load models larger than 8GB without linking multiple cards, a complex setup with diminishing returns.

Ultimately, the benchmark shows there's no single 'best' card, but a clear choice based on use case. For running 7B-14B models efficiently, the RTX 3060 or CMP100-210 offer the best blend of speed and cost. For developers needing to experiment with larger 20B+ models on a tight budget, the Tesla P40's 24GB VRAM is the only viable sub-$250 option, despite its older architecture and higher power draw.