Reddit user recalls 20-min token generation with Bloom on 768GB optane drive
One token every 20 minutes—early local AI was painfully slow
Deep Dive
A Reddit user reminisces about buying 768GB Optane drives to run the full Bloom model with swap, waiting 20 minutes per token, and asks if others played with early models. They want to revisit them to see how much progress has happened locally.
Key Points
- Bloom (176B parameters) required 768GB optane drives and swap to run locally
- Early users reported 20-minute latency per token generation
- Modern 8B-parameter models now run in milliseconds on consumer hardware
Why It Matters
Illustrates the raw progress in AI inference—from minutes per token to milliseconds per token