New Study Shows AI's Hidden Energy Costs: Longer Answers Mean Bigger Bills

When you ask an AI a question, it burns electricity—and lots of it. Until now, most people focused on a simple metric: energy per token. A token is basically a chunk of a word, so think of it as energy per word. That metric looks great: as the AI writes a longer answer, each individual word becomes cheaper to produce. But a new study from Carnegie Mellon researchers reveals the trick: the total energy for the entire request actually goes up. You're just spreading a bigger energy bill across more words.

The study tested powerful Nvidia H100 and H200 GPUs—the chips that run most AI services. They found that if an AI generates a 10-word answer, it uses about 7.5 joules per word. But if it writes a 512-word essay, it uses just 0.72 joules per word. Sounds efficient, right? Except the whole answer now gobbles up nearly five times more total electricity. That's like buying in bulk: each item costs less, but you spend more at checkout.

The researchers also looked at batching—putting many users' requests through the same GPU at once. Batching helps a lot for short conversations, cutting energy per request by over six times. But for long, detailed prompts, that advantage nearly disappears. And newer "mixture of experts" models, which only activate parts of their brain for each task, actually make this problem worse. They use more energy at the start of each request, though batching can narrow the gap.

What does this mean for you? AI companies pay for electricity, and those costs eventually show up in subscription prices. Data centers also need more cooling and power, straining local grids and increasing carbon emissions. The study concludes that companies should stop obsessing over per-word efficiency and start measuring the whole picture. Until they do, AI's environmental and financial costs may keep climbing—even while headlines promise cheaper, faster models.