New Study: 'Explainable' AI Rules Don't Really Explain the AI

When an AI learns by trial and error (called reinforcement learning — the same idea as training a dog with treats), it usually gets handed to us as a sealed black box. You can see that it was trained, but not what it actually learned. One popular fix is to extract simple written rules — 'if the pole leans left, push left' — and hand those to humans as an explanation. The new paper puts that idea under a microscope, and the results are a warning for anyone who trusts a rule sheet as proof an AI is safe.

Using two classic test games, CartPole (balancing a pole on a cart) and Acrobot (swinging a two-jointed arm to a target), the author trained many AI versions and pulled rules out of each. The key finding: sharing rules does not mean sharing behavior. Two systems can list the same rules and still make completely different choices in situations they haven't seen before. In plain terms, matching rulebooks are not matching drivers.

The paper also tested whether you can merge two trained AIs into one that's smarter than both — like combining two specialists into a better teammate. It doesn't really work. The merged system just picks between rules the original AIs already had; it doesn't invent a new skill. One apparent success turned out to be a measurement error, and when the author corrected how the rules were tested, the result flipped. Even a standard sanity check for whether the merge beat a simpler method failed in both games.

So what should you take away? Auditing AI isn't a single stamp of approval. The author proposes six separate, testable questions — including whether the rules are even accurate, whether they cover real situations, and whether they hold up in practice. It's a call for honesty: if a company says its AI is 'explainable,' ask which of the six it actually proved, and which it quietly skipped.