Why Accurate AI Can Still Get Cause and Effect Wrong

When someone asks a big question like "does remote work raise productivity?" or "does this drug save lives?", data alone can be misleading. Other factors — age, income, habits — get in the way. Statisticians use helper models to clear out those outside factors before estimating the real causal effect. These helpers are called nuisance functions. The usual way to check them is prediction error: how well did the model fit known data? This paper asks a sharp question: does a better-fitting helper actually lead to a better causal answer?

Using simulated data, the researcher compared four methods: simple regression, flexible statistical models, XGBoost (a popular machine learning tool), and double machine learning with XGBoost. XGBoost usually had the lowest prediction error and the best point estimates among the non-expert methods. But double machine learning gave confidence intervals that covered the true answer more reliably. The key finding: prediction error did not consistently track causal bias. A model could nail its predictions and still produce a causal conclusion that was off.

Why does this matter? Companies, hospitals, and governments increasingly rely on machine learning to decide which actions actually work. Model reports often brag about prediction accuracy. If that accuracy doesn't reflect whether the underlying cause-and-effect claim is correct, decisions can go wrong in invisible ways. The paper also tried a combined error measure involving both helper models, but it was too weak to signal causal quality.

None of this means prediction accuracy is useless. It means prediction is not proof. Causal estimates need their own validation: tests of bias, uncertainty, and confidence coverage. The lesson is simple and important: check the answer, not just the model's report card.