Why AI Chatbots Are Getting Faster (And Costlier)
Your AI assistant’s speed and your phone’s battery life are locked in a high-stakes battle
A review of AI hardware accelerators for large language models finds the real bottleneck isn't raw compute—it's memory. The transformer's autoregressive decoding is bandwidth-bound, the key-value cache can rival model weights in size, and data movement dominates energy use. No single architecture is optimal everywhere: GPUs remain the flexible default and workhorse for training, domain-specific ASICs win at scale for stable, high-volume workloads, processing-in-memory looks like the most promising near-term response to the memory wall, and neuromorphic and photonic approaches aren't production-ready yet. The path forward, according to the article, is hardware-algorithm co-design and heterogeneous, memory-centric systems—because for large language models, the memory system has become the computer.
- AI chatbots are hitting a 'memory wall,' slowing them down and making them more expensive to run.
- No single computer chip is perfect for all AI tasks—GPUs are flexible but not efficient, while specialized chips are fast but costly.
- New 'processing-in-memory' chips could help, but they’re not ready yet and may increase energy use and prices.
Why It Matters
Faster AI chatbots could save you time, but the hidden tech battle may mean higher costs or drained phone batteries.