New Study Argues LLM Explainability Should Be About Practice, Not Peeking Inside

Researchers Marvin Pafla, Jesse Hoey, Kate Larson, and Mark Hancock from the University of Waterloo published a provocative paper arguing that current explainable AI (XAI) methods for large language models are fundamentally flawed. Drawing on Dourish's embodied interaction theory and enactivist cognition, they contend that understanding is created in use—not extracted from a model's internals. They identify two 'ontological obstacles': first, surrogate explanations (like attention maps or feature attributions) import external abstractions that can be mistaken for the model's actual reasoning; second, focusing on internal mechanisms ignores that explainers themselves participate in creating understanding. The authors argue this misdirection can increase overreliance on LLM outputs.

Rather than treating explainability as a property of the model, the paper proposes an embodied approach where explanations are designed as affordances for action in shared practice. This means explanations should let users probe, coordinate, and repair behavior in real-world contexts—not just inspect internal weights or activations. The team argues that many current XAI outputs are misnamed; they are not true explanations but rather 'surrogates' that can mislead. For professionals building LLM-powered systems, this suggests a shift from debugging model internals to designing interfaces that support collaborative sense-making between human and machine.