Caltech: Better AI Forecasts Can Backfire When Opponents Adapt

Artificial intelligence is increasingly used to guide real decisions, not just make predictions. A park ranger might use AI to guess where endangered animals are hiding, then send patrols there. A utility might use AI to guess where an attack is likely, then guard that spot. Normally, everyone assumes that a more accurate forecast is automatically better. A new paper from researchers at Caltech says that assumption breaks down the moment another person is playing against you.

Here's the catch. Poachers aren't passive. If the AI predicts wildlife will be in one valley and patrols go there, the poachers simply go somewhere else. The original prediction may have been perfectly correct — the animals really were there — but the poachers moved, so the patrols caught nothing. In other words, the better the AI got at predicting, the more effectively it taught the poachers what to avoid. The researchers found the relationship between accuracy and real-world results isn't a straight line; it can actually go backward.

To fix this, the team developed new algorithms that train AI to care about the final outcome, not just the accuracy of its guesses. Instead of asking "where are the animals?" the system asks "which patrol pattern leads to the fewest poached animals?" They tested their approach in two settings: wildlife conservation and protecting infrastructure like power grids. In both cases, accounting for how the other side reacts produced better real-world results than simply chasing accurate predictions.

So why should you care? The same logic applies far beyond rangers and poachers. Fraud detection, tax audits, traffic enforcement, security screening, and even retail pricing all involve people who adjust once they figure out the system. If a bank's AI flags suspicious transactions perfectly, criminals adapt and the fraud just moves. This paper is a reminder that AI is rarely a one-player game — and that the scoreboard should measure outcomes, not predictions.