Tech Tinkerer Runs Massive AI on a Small Device, But at a Crawl
This shows anyone can run powerful AI at home—but it's painfully slow.
Someone tracked their first run of MiMo Pro on a Jetson AGX Thor at a scorching 0.646 tokens/sec — but it did technically load and run, so it counts. Every decoder layer, routed expert and SSD read was traced and replayable step by step.
The model is roughly 566GB in FP8, read from disk mid-inference and pushed through 128GB of memory, which the author compares to the speed and architecture of hand-extruded hotdog manufacturing. They figure it's probably fine but decided to take a closer look to see if they could reach factory hotdog manufacturing speeds — "like, a small, local hotdog factory" — joking along the way about Dario, a case of MiMos that might last through Halloween, and signing off with "get well soon Dario!"
- Open-source AI models let you see and modify the code, unlike closed models from companies like OpenAI.
- A 566GB AI model ran on a 128GB device by streaming from disk, but at 0.646 tokens per second—about 100x slower than typical chatbots.
- This proves powerful AI can run on small, local hardware, but speed and cost are still barriers for everyday use.
Why It Matters
Open-source AI gives you transparency and control, but running big models locally is still slow and techy.