AI's 'Explainable' Feature Is Flawed, Study Finds

A recent audit of a technique called VPD (adversarial Parameter Decomposition) has uncovered systematic problems in how well it can explain AI decisions. VPD aims to break down an AI model into simple parts that can be switched off without affecting the output, which is crucial for understanding and trusting AI. The study found that when an 'adversary' (a program that searches for weaknesses) tries to break the model, the defects it finds are not just random quirks but carry over to new, unseen inputs. This means the flaws are consistent and could undermine the reliability of AI explanations.

The audit tested a small language model and found that after only 20 steps of searching, the adversary could find sets of components that, when turned off, changed the model's output in ways that also affected other inputs. This suggests that the method's training process isn't preventing these systematic defects. The authors suggest a practical improvement to the training that could help, but the current version is still imperfect.

For everyday people, this might seem distant, but it matters because AI is increasingly used in decisions that affect our lives—from loan approvals to medical diagnoses. If we can't trust the explanations of why an AI made a certain choice, it's harder to hold systems accountable and ensure they're fair. This research highlights that we're still far from having AI that can truly explain itself in a robust way.

While the VPD authors have made improvements in a newer, unpublished version, the publicly released version still has these issues. The study emphasizes the need for continued scrutiny and better methods for making AI transparent.